Sep 16 18:44:52.439765 ip-10-0-143-161 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:44:52.842252 ip-10-0-143-161 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:52.842252 ip-10-0-143-161 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:44:52.842252 ip-10-0-143-161 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:52.842252 ip-10-0-143-161 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:44:52.842252 ip-10-0-143-161 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:52.843659 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.843543    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:44:52.846452 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846436    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:52.846452 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846452    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846456    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846459    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846462    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846465    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846469    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846472    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846486    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846489    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846492    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846495    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846498    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846500    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846503    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846506    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846509    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846512    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846514    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846517    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:52.846528 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846524    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846527    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846530    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846532    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846535    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846538    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846541    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846543    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846546    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846549    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846553    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846556    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846558    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846561    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846563    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846566    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846568    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846571    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846573    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846575    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:52.846974 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846578    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846580    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846583    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846585    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846588    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846591    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846593    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846596    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846600    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846604    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846606    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846609    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846611    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846614    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846617    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846620    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846623    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846625    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846628    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846630    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:52.847502 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846633    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846635    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846639    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846643    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846647    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846650    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846653    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846656    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846658    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846661    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846663    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846666    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846672    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846674    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846677    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846681    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846684    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846686    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846689    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:52.847984 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846691    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846694    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846696    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846699    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846701    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846704    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846706    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846710    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846713    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846716    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846718    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846721    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846723    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846726    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846728    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846731    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846734    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846736    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846739    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.846741    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:52.848443 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847443    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847449    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847452    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847456    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847459    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847462    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847465    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847468    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847471    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847492    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847495    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847498    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847500    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847503    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847506    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847509    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847512    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847515    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847518    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:52.848957 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847521    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847524    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847527    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847530    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847532    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847535    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847537    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847540    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847543    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847546    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847549    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847551    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847554    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847556    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847559    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847562    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847564    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847567    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847569    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:52.849439 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847573    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847576    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847578    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847581    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847583    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847586    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847588    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847591    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847593    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847596    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847599    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847602    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847605    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847608    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847611    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847614    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847616    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847619    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847621    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847624    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:52.849951 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847626    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847629    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847632    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847634    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847639    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847642    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847645    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847648    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847651    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847653    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847656    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847659    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847662    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847665    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847667    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847669    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847672    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847675    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847678    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847681    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:52.850454 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847683    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847687    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847690    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847694    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847697    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847700    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847703    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847705    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847708    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847711    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847713    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847716    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847718    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847721    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847725    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847729    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847732    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847735    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847738    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847741    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:52.850952 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.847744    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848129    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848134    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848138    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848140    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848143    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848146    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848149    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848151    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848154    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848157    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848159    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848162    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848164    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848167    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848170    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848173    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848176    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848181    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848184    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:52.851447 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848187    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848190    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848192    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848195    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848199    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848202    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848205    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848207    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848209    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848212    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848214    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848217    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848220    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848222    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848225    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848227    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848230    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848232    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848235    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:52.851949 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848238    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848240    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848242    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848245    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848248    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848250    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848253    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848256    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848259    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848261    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848264    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848267    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848269    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848271    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848274    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848280    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848282    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848285    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848287    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848290    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:52.852408 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848292    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848295    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848297    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848300    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848302    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848304    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848307    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848310    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848314    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848316    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848319    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848322    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848325    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848328    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848331    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848333    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848336    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848339    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848341    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848344    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:52.852931 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848347    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848350    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848352    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848355    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848357    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848360    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848364    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848398    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848402    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848405    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848408    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848411    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848414    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848417    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848420    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848423    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848426    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848428    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848431    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848434    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:52.853413 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.848436    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848522    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848560    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848569    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848574    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848579    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848582    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848586    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848591    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848594    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848597    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848601    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848604    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848608    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848611    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848614    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848617    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848620    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848623    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848628    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848631    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848635    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848638    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848642    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:44:52.853957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848646    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848649    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848652    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848655    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848659    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848662    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848665    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848668    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848671    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848675    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848678    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848681    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848684    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848687    2578 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848691    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848695    2578 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848698    2578 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848701    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848704    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848709    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848713    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848716    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848719    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848722    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848725    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:44:52.854567 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848728    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848731    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848734    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848737    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848740    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848743    2578 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848747    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848750    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848753    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848757    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848760    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848763    2578 flags.go:64] FLAG: --help="false"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848766    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-143-161.ec2.internal"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848769    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848772    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848775    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848778    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848781    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848784    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848787    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848790    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848793    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848796    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848800    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:44:52.855188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848803    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848805    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848809    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848813    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848816    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848818    2578 flags.go:64] FLAG: --lock-file=""
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848821    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848824    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848827    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848833    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848836    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848839    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848842    2578 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848845    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848849    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848852    2578 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848855    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848859    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848862    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848867    2578 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848870    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848873    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848876    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848879    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848882    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:44:52.855797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848885    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848888    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848895    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848898    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848901    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848904    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848908    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848913    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848916    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848920    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848924    2578 flags.go:64] FLAG: --port="10250"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848928    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848930    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05f2441e0fb19d43f"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848934    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848937    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848939    2578 flags.go:64] FLAG: --register-node="true"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848942    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848946    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848949    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848952    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848955    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848959    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848962    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848966    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848969    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:44:52.856426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848972    2578 flags.go:64] FLAG: --runonce="false"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848976    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848979    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848982    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848985    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848988    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848992    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848995    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.848998    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849001    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849004    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849007    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849010    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849013    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849017    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849020    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849025    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849030    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849033    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849038    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849041    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849043    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849046    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849050    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849054    2578 flags.go:64] FLAG: --v="2"
Sep 16 18:44:52.857245 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849058    2578 flags.go:64] FLAG: --version="false"
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849062    2578 flags.go:64] FLAG: --vmodule=""
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849066    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.849069    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849210    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849216    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849220    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849223    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849227    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849230    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849233    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849236    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849238    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849241    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849244    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849247    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849249    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849252    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849255    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849257    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:52.858216 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849260    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849262    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849266    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849269    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849273    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849276    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849280    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849283    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849286    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849289    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849291    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849296    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849298    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849301    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849303    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849306    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849309    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849312    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849314    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:52.858751 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849317    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849320    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849323    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849326    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849328    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849331    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849333    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849336    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849338    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849341    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849344    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849346    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849349    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849352    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849354    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849357    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849360    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849364    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849367    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849370    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:52.859255 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849372    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849375    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849378    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849380    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849384    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849387    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849390    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849392    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849395    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849398    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849400    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849404    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849407    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849410    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849413    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849416    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849419    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849421    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849424    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:52.859796 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849426    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849429    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849431    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849434    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849436    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849439    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849442    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849444    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849447    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849450    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849455    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849457    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849460    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849463    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849465    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849468    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849471    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849486    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849489    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849491    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:52.860267 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849494    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849497    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849500    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849503    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.849505    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.850169    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.858595    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.858740    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.858762    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.858771    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858831    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858836    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858840    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858843    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858847    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:52.860772 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858850    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858853    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858856    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858860    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858862    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858866    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858868    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858871    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858874    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858877    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858880    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858882    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858885    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858888    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858890    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858894    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858898    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858900    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858903    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858906    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:52.861142 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858908    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858911    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858914    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858918    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858921    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858924    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858927    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858930    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858933    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858935    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858938    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858941    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858943    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858946    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858949    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858951    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858954    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858957    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858959    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:52.861643 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858962    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858965    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858968    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858970    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858973    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858975    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858978    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858981    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858984    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858986    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858989    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858992    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858995    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.858998    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859001    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859003    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859006    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859009    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859011    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859014    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:52.862153 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859017    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859019    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859023    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859027    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859030    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859033    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859036    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859039    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859042    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859045    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859048    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859051    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859054    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859062    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859066    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859071    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859074    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859077    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859080    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:52.862657 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859083    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859086    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859089    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859092    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859095    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859098    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859100    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859103    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859106    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859108    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859111    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859114    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859117    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859119    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859122    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:52.863160 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859125    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.859130    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859251    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859256    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859260    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859263    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859266    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859269    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859271    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859274    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859277    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859280    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859282    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859285    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859288    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:52.863544 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859291    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859294    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859296    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859299    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859302    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859304    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859307    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859310    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859312    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859315    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859318    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859322    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859326    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859329    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859332    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859336    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859339    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859342    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859345    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:52.863917 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859348    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859351    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859353    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859356    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859359    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859361    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859364    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859367    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859370    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859372    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859375    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859377    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859380    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859383    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859385    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859388    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859390    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859393    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859395    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859398    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:52.864382 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859400    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859403    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859406    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859408    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859411    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859413    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859416    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859419    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859421    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859424    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859427    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859429    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859432    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859434    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859437    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859440    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859442    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859445    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859447    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859450    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:52.864994 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859452    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859455    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859457    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859460    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859462    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859465    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859468    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859470    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859473    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859489    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859492    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859495    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859497    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859500    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859503    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859505    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859508    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859511    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859514    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:52.865521 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859516    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859520    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859527    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859530    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859533    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859536    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859539    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:52.859542    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.859547    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.859676    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.861752    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.862689    2578 server.go:1019] "Starting client certificate rotation"
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.862784    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:52.865982 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.862820    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:52.866318 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.865259    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:52.866434 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.866417    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:52.876785 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.876769    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:44:52.879705 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.879687    2578 log.go:25] "Validated CRI v1 image API"
Sep 16 18:44:52.881772 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.881759    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:44:52.887847 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.887826    2578 fs.go:135] Filesystem UUIDs: map[0d500161-4f7b-4552-94d9-0a323b9c6b22:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7cd0864d-c018-4035-ab39-b46e4f03e6a6:/dev/nvme0n1p4]
Sep 16 18:44:52.887931 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.887845    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:44:52.894426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.894301    2578 manager.go:216] Machine: {Timestamp:2026-09-16 18:44:52.892081787 +0000 UTC m=+0.339176320 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099800 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26d1c8fac4f9a07b48794ef9b6535e SystemUUID:ec26d1c8-fac4-f9a0-7b48-794ef9b6535e BootID:8c7b3dfa-ca96-40c2-820a-26d205467a26 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:b1:72:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:b1:72:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:06:15:2c:99:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:44:52.894426 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.894418    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:44:52.894578 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.894523    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:44:52.895558 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.895542    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:52.896848 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.896825    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:44:52.896986 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.896853    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:44:52.897582 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.897569    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:44:52.897626 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.897592    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:44:52.897626 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.897603    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:52.897626 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.897621    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:52.898584 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.898574    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:44:52.902591 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.902579    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:52.902716 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.902706    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:44:52.902747 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.902736    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:44:52.902777 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.902750    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:44:52.902777 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.902766    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:44:52.902777 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.902776    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:44:52.902881 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.902793    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:44:52.903252 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.903232    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:52.903357 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.903266    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:52.909992 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.909970    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:44:52.912269 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.912256    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:44:52.912317 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.912287    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:44:52.913741 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913727    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:44:52.913822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913745    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:44:52.913822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913752    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:44:52.913822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913759    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:44:52.913822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913771    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:44:52.913822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913780    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:44:52.913822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913794    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:44:52.913822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913799    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:44:52.914001 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913833    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:44:52.914001 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913840    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:44:52.914001 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913849    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:44:52.914001 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.913855    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:44:52.914565 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.914554    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:44:52.914596 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.914568    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:44:52.918620 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.918596    2578 server.go:1268] "Started kubelet"
Sep 16 18:44:52.918723 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.918674    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-161.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:44:52.918790 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.918739    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:44:52.918790 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.918745    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:44:52.918889 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.918752    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:44:52.918889 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.918814    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:44:52.918889 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.918839    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:44:52.919627 ip-10-0-143-161 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:44:52.920107 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.920090    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:44:52.920308 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.920296    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:44:52.923182 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.923146    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:44:52.925535 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.925518    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:44:52.925621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.925538    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:52.925728 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.924816    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-161.ec2.internal.18d5e16ecb079f2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-161.ec2.internal,UID:ip-10-0-143-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-161.ec2.internal,},FirstTimestamp:2026-09-16 18:44:52.918566701 +0000 UTC m=+0.365661222,LastTimestamp:2026-09-16 18:44:52.918566701 +0000 UTC m=+0.365661222,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-161.ec2.internal,}"
Sep 16 18:44:52.925813 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.925759    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:52.926097 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.926080    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:44:52.926167 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.926123    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:44:52.926167 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.926143    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:44:52.927220 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.926573    2578 factory.go:55] Registering systemd factory
Sep 16 18:44:52.927220 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.926586    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:44:52.927322 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.927292    2578 factory.go:153] Registering CRI-O factory
Sep 16 18:44:52.927322 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.927310    2578 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:44:52.927413 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.927362    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:44:52.927413 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.927383    2578 factory.go:103] Registering Raw factory
Sep 16 18:44:52.927413 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.927400    2578 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:44:52.928006 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.927992    2578 manager.go:349] Starting recovery of all containers
Sep 16 18:44:52.928066 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.928052    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:44:52.928066 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.928062    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:44:52.932319 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.932295    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hn995"
Sep 16 18:44:52.934457 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.934425    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:44:52.938702 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.938677    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hn995"
Sep 16 18:44:52.938702 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.938675    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:44:52.938972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.938958    2578 manager.go:354] Recovery completed
Sep 16 18:44:52.943778 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.943761    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:52.945768 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.945752    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:52.945855 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.945786    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:52.945855 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.945801    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:52.946455 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.946440    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:44:52.946455 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.946450    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:52.946599 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.946451    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:44:52.946599 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.946507    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:52.947290 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.947270    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:52.947342 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.947309    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:52.947342 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.947324    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:52.948257 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.948136    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-161.ec2.internal.18d5e16ecca6b120  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-161.ec2.internal,UID:ip-10-0-143-161.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-161.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-161.ec2.internal,},FirstTimestamp:2026-09-16 18:44:52.945768736 +0000 UTC m=+0.392863259,LastTimestamp:2026-09-16 18:44:52.945768736 +0000 UTC m=+0.392863259,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-161.ec2.internal,}"
Sep 16 18:44:52.948686 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.948669    2578 policy_none.go:49] "None policy: Start"
Sep 16 18:44:52.948686 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.948686    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:44:52.948792 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.948696    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:44:52.950978 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.950965    2578 policy_none.go:47] "Start"
Sep 16 18:44:52.974167 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.974149    2578 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:44:52.974287 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.974193    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:44:52.974287 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.974206    2578 server.go:87] "Starting device plugin registration server"
Sep 16 18:44:52.974608 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.974594    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:44:52.974653 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.974619    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:44:52.974720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.974708    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:44:52.975124 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.974781    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:44:52.975124 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:52.974792    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:44:52.975449 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.975425    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:44:52.975559 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:52.975495    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:53.056033 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.055987    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:44:53.058018 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.057180    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:44:53.058018 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.057227    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:44:53.058018 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.057252    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:44:53.058018 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.057288    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:44:53.060014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.059997    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:53.075136 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.075114    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:53.076180 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.076164    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:53.076268 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.076194    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:53.076268 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.076205    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:53.076268 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.076254    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.084931 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.084914    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.085016 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.084939    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-161.ec2.internal\": node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:53.101236 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.101163    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:53.157646 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.157611    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal"]
Sep 16 18:44:53.157792 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.157693    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:53.158772 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.158756    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:53.158845 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.158787    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:53.158845 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.158802    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:53.161104 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.161090    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:53.161230 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.161215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.161288 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.161245    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:53.161908 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.161891    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:53.162002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.161917    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:53.162002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.161926    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:53.162002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.161941    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:53.162002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.161966    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:53.162002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.161976    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:53.164072 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.164056    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.164154 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.164081    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:53.164868 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.164840    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:53.164868 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.164869    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:53.165019 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.164885    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:53.193401 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.193370    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-161.ec2.internal\" not found" node="ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.197735 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.197719    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-161.ec2.internal\" not found" node="ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.201768 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.201754    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:53.229467 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.229441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae49c932e0b671a70c1b9b1effab98ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal\" (UID: \"ae49c932e0b671a70c1b9b1effab98ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.229592 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.229471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae49c932e0b671a70c1b9b1effab98ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal\" (UID: \"ae49c932e0b671a70c1b9b1effab98ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.229592 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.229505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/240b210d213d06d9b2a5c1727b8ee1a0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-161.ec2.internal\" (UID: \"240b210d213d06d9b2a5c1727b8ee1a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.301874 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.301845    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:53.330182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.330157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae49c932e0b671a70c1b9b1effab98ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal\" (UID: \"ae49c932e0b671a70c1b9b1effab98ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.330275 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.330187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae49c932e0b671a70c1b9b1effab98ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal\" (UID: \"ae49c932e0b671a70c1b9b1effab98ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.330275 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.330204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/240b210d213d06d9b2a5c1727b8ee1a0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-161.ec2.internal\" (UID: \"240b210d213d06d9b2a5c1727b8ee1a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.330275 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.330243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/240b210d213d06d9b2a5c1727b8ee1a0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-161.ec2.internal\" (UID: \"240b210d213d06d9b2a5c1727b8ee1a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.330275 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.330253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae49c932e0b671a70c1b9b1effab98ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal\" (UID: \"ae49c932e0b671a70c1b9b1effab98ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.330407 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.330256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae49c932e0b671a70c1b9b1effab98ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal\" (UID: \"ae49c932e0b671a70c1b9b1effab98ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.402524 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.402421    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:53.500046 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.500002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.500155 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.500060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.503499 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.503429    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:53.606886 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.606845    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:53.712427 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.712333    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-161.ec2.internal\" not found"
Sep 16 18:44:53.773369 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.773344    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:53.826884 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.826842    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.839142 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.839112    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:53.841372 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.841350    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal"
Sep 16 18:44:53.848717 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.848693    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:53.862879 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.862851    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:44:53.863017 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.862999    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:53.863058 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.863006    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:53.903665 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.903632    2578 apiserver.go:52] "Watching apiserver"
Sep 16 18:44:53.926491 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.926451    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:53.929785 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.929758    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:44:53.930138 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.930114    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rfcc8","kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8whgg","openshift-dns/node-resolver-9pgsd","openshift-image-registry/node-ca-zh85g","openshift-multus/multus-additional-cni-plugins-r2949","openshift-multus/multus-srgnm","openshift-multus/network-metrics-daemon-6fpgx","kube-system/konnectivity-agent-zhnkf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal","openshift-network-diagnostics/network-check-target-n5x8n","openshift-network-operator/iptables-alerter-k2xhd","openshift-ovn-kubernetes/ovnkube-node-knlxg"]
Sep 16 18:44:53.932775 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.932756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:53.932862 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.932840    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:44:53.934882 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.934867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:53.937082 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.937064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:53.938421 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.938402    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8k2wh\""
Sep 16 18:44:53.938527 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.938405    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:53.938527 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.938452    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:53.939254 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.939238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:53.941575 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.941559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:53.943829 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.943802    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-srgnm"
Sep 16 18:44:53.944508 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.944471    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:39:53 +0000 UTC" deadline="2028-06-09 21:28:44.449568488 +0000 UTC"
Sep 16 18:44:53.944508 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.944508    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15170h43m50.505064744s"
Sep 16 18:44:53.946025 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.946008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:53.946086 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.946070    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:44:53.948204 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:44:53.948515 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948492    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:44:53.948671 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:44:53.948671 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948578    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:44:53.948671 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948618    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:44:53.948671 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948630    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:44:53.948671 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948641    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:44:53.948671 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948647    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:44:53.949012 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948757    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:44:53.949012 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948818    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:44:53.949012 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948876    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cndmf\""
Sep 16 18:44:53.949012 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.948915    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:44:53.949204 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.949190    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:44:53.949266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.949213    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-chxht\""
Sep 16 18:44:53.950575 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.950558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:53.950643 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.950584    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gxfwj\""
Sep 16 18:44:53.952122 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.952098    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:44:53.952291 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.952279    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7gjdq\""
Sep 16 18:44:53.952926 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.952911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:44:53.953008 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:53.952963    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:44:53.953698 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.953681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xj8nj\""
Sep 16 18:44:53.954904 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.954886    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:44:53.955312 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.955296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:53.956462 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.956442    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:53.956750 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.956730    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:44:53.956830 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.956735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:44:53.956830 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.956776    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:44:53.956830 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.956800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9rmr\""
Sep 16 18:44:53.957811 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.957792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:53.958357 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.958342    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:53.958425 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.958374    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:44:53.958992 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.958977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:53.959054 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.959005    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fgdr2\""
Sep 16 18:44:53.967605 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.967539    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:44:53.967698 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.967619    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:44:53.970067 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.970051    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:44:53.970668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.970653    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:44:53.970757 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.970742    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:44:53.970809 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.970657    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:44:53.975219 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:53.975200    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8xlg\""
Sep 16 18:44:54.006761 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.006736    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-998z4"
Sep 16 18:44:54.018728 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.018702    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-998z4"
Sep 16 18:44:54.027234 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.027216    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:44:54.034370 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjc9\" (UniqueName: \"kubernetes.io/projected/acb7ae1c-13d5-4745-aa4d-8e5f951e8f15-kube-api-access-vpjc9\") pod \"node-ca-zh85g\" (UID: \"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15\") " pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.034532 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-sys-fs\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.034532 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-slash\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.034532 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc784\" (UniqueName: \"kubernetes.io/projected/e943c3d7-af76-4889-9ab2-a272b6aef96a-kube-api-access-vc784\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.034710 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ws5\" (UniqueName: \"kubernetes.io/projected/384ea14e-b6d2-4db5-8b15-d3990b168f06-kube-api-access-f5ws5\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.034710 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-cnibin\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.034710 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b181e08-247f-4e86-bf19-6a71d5d9edeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.034710 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7cpk\" (UniqueName: \"kubernetes.io/projected/1b181e08-247f-4e86-bf19-6a71d5d9edeb-kube-api-access-b7cpk\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.034710 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-socket-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.034710 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-systemd-units\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.034710 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-run-ovn-kubernetes\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.034972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-run\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.034972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-device-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.034972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc6a0944-4d3b-49ad-9f8f-6b438e040378-cni-binary-copy\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.034972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2a79614-aba4-44dd-928e-ccb32df6e0d0-iptables-alerter-script\") pod \"iptables-alerter-k2xhd\" (UID: \"a2a79614-aba4-44dd-928e-ccb32df6e0d0\") " pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.034972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.034972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-registration-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.034972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.034972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-system-cni-dir\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.034994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-var-lib-kubelet\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-run-multus-certs\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acb7ae1c-13d5-4745-aa4d-8e5f951e8f15-host\") pod \"node-ca-zh85g\" (UID: \"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15\") " pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-sys\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/384ea14e-b6d2-4db5-8b15-d3990b168f06-tmp\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7f9ed7d-0ea5-4c26-b542-703da8f84edb-hosts-file\") pod \"node-resolver-9pgsd\" (UID: \"b7f9ed7d-0ea5-4c26-b542-703da8f84edb\") " pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70f46be3-8082-4906-8168-ca1dff840646-agent-certs\") pod \"konnectivity-agent-zhnkf\" (UID: \"70f46be3-8082-4906-8168-ca1dff840646\") " pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-kubelet\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-run-netns\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.035266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-run-openvswitch\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e943c3d7-af76-4889-9ab2-a272b6aef96a-ovnkube-config\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ce0de65-1777-4f99-ae24-df43d786f881-kubelet-config\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-sysconfig\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f9ed7d-0ea5-4c26-b542-703da8f84edb-tmp-dir\") pod \"node-resolver-9pgsd\" (UID: \"b7f9ed7d-0ea5-4c26-b542-703da8f84edb\") " pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhjl\" (UniqueName: \"kubernetes.io/projected/a2a79614-aba4-44dd-928e-ccb32df6e0d0-kube-api-access-gvhjl\") pod \"iptables-alerter-k2xhd\" (UID: \"a2a79614-aba4-44dd-928e-ccb32df6e0d0\") " pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b181e08-247f-4e86-bf19-6a71d5d9edeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwkn\" (UniqueName: \"kubernetes.io/projected/48993031-1a2a-4648-a040-79f0c8c09a1e-kube-api-access-6nwkn\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-run-ovn\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-kubernetes\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70f46be3-8082-4906-8168-ca1dff840646-konnectivity-ca\") pod \"konnectivity-agent-zhnkf\" (UID: \"70f46be3-8082-4906-8168-ca1dff840646\") " pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b181e08-247f-4e86-bf19-6a71d5d9edeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-systemd\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-conf-dir\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b181e08-247f-4e86-bf19-6a71d5d9edeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.035764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ce0de65-1777-4f99-ae24-df43d786f881-dbus\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-socket-dir-parent\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-hostroot\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-etc-kubernetes\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-log-socket\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-run-netns\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-cni-bin\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-cni-netd\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-daemon-config\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.035990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b181e08-247f-4e86-bf19-6a71d5d9edeb-cnibin\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-var-lib-openvswitch\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e943c3d7-af76-4889-9ab2-a272b6aef96a-ovnkube-script-lib\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-run-k8s-cni-cncf-io\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5sqf\" (UniqueName: \"kubernetes.io/projected/fc6a0944-4d3b-49ad-9f8f-6b438e040378-kube-api-access-x5sqf\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.036253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b181e08-247f-4e86-bf19-6a71d5d9edeb-system-cni-dir\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-modprobe-d\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-lib-modules\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-host\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-cni-dir\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-node-log\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e943c3d7-af76-4889-9ab2-a272b6aef96a-env-overrides\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e943c3d7-af76-4889-9ab2-a272b6aef96a-ovn-node-metrics-cert\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-var-lib-kubelet\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-tuned\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2a79614-aba4-44dd-928e-ccb32df6e0d0-host-slash\") pod \"iptables-alerter-k2xhd\" (UID: \"a2a79614-aba4-44dd-928e-ccb32df6e0d0\") " pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acb7ae1c-13d5-4745-aa4d-8e5f951e8f15-serviceca\") pod \"node-ca-zh85g\" (UID: \"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15\") " pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8r8\" (UniqueName: \"kubernetes.io/projected/dd44d99b-8aab-441b-a376-af9a0d55f130-kube-api-access-rb8r8\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-run-systemd\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g5pd\" (UniqueName: \"kubernetes.io/projected/b7f9ed7d-0ea5-4c26-b542-703da8f84edb-kube-api-access-2g5pd\") pod \"node-resolver-9pgsd\" (UID: \"b7f9ed7d-0ea5-4c26-b542-703da8f84edb\") " pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-var-lib-cni-bin\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-etc-openvswitch\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.036825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-sysctl-d\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.037621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-sysctl-conf\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.037621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.036604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-var-lib-cni-multus\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.124591 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.124538    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:54.136874 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.136793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-lib-modules\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.137034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.136910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-host\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.137034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.136936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-cni-dir\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.137034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.136961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-node-log\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.137034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.136980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-lib-modules\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.137034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.136986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e943c3d7-af76-4889-9ab2-a272b6aef96a-env-overrides\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.137034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-host\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e943c3d7-af76-4889-9ab2-a272b6aef96a-ovn-node-metrics-cert\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-node-log\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-var-lib-kubelet\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-cni-dir\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-tuned\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2a79614-aba4-44dd-928e-ccb32df6e0d0-host-slash\") pod \"iptables-alerter-k2xhd\" (UID: \"a2a79614-aba4-44dd-928e-ccb32df6e0d0\") " pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acb7ae1c-13d5-4745-aa4d-8e5f951e8f15-serviceca\") pod \"node-ca-zh85g\" (UID: \"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15\") " pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8r8\" (UniqueName: \"kubernetes.io/projected/dd44d99b-8aab-441b-a376-af9a0d55f130-kube-api-access-rb8r8\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-var-lib-kubelet\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2a79614-aba4-44dd-928e-ccb32df6e0d0-host-slash\") pod \"iptables-alerter-k2xhd\" (UID: \"a2a79614-aba4-44dd-928e-ccb32df6e0d0\") " pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-run-systemd\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5pd\" (UniqueName: \"kubernetes.io/projected/b7f9ed7d-0ea5-4c26-b542-703da8f84edb-kube-api-access-2g5pd\") pod \"node-resolver-9pgsd\" (UID: \"b7f9ed7d-0ea5-4c26-b542-703da8f84edb\") " pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.137326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-var-lib-cni-bin\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-etc-openvswitch\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-sysctl-d\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-sysctl-conf\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-var-lib-cni-multus\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-run-systemd\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjc9\" (UniqueName: \"kubernetes.io/projected/acb7ae1c-13d5-4745-aa4d-8e5f951e8f15-kube-api-access-vpjc9\") pod \"node-ca-zh85g\" (UID: \"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15\") " pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-sys-fs\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-var-lib-cni-bin\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-slash\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-sys-fs\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-var-lib-cni-multus\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-sysctl-d\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-sysctl-conf\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-etc-openvswitch\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc784\" (UniqueName: \"kubernetes.io/projected/e943c3d7-af76-4889-9ab2-a272b6aef96a-kube-api-access-vc784\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-slash\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acb7ae1c-13d5-4745-aa4d-8e5f951e8f15-serviceca\") pod \"node-ca-zh85g\" (UID: \"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15\") " pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.138037 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ws5\" (UniqueName: \"kubernetes.io/projected/384ea14e-b6d2-4db5-8b15-d3990b168f06-kube-api-access-f5ws5\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-cnibin\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e943c3d7-af76-4889-9ab2-a272b6aef96a-env-overrides\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b181e08-247f-4e86-bf19-6a71d5d9edeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7cpk\" (UniqueName: \"kubernetes.io/projected/1b181e08-247f-4e86-bf19-6a71d5d9edeb-kube-api-access-b7cpk\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-cnibin\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-socket-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-systemd-units\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137972    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.137989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-run-ovn-kubernetes\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-run\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-run-ovn-kubernetes\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-systemd-units\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-socket-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-device-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-run\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc6a0944-4d3b-49ad-9f8f-6b438e040378-cni-binary-copy\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2a79614-aba4-44dd-928e-ccb32df6e0d0-iptables-alerter-script\") pod \"iptables-alerter-k2xhd\" (UID: \"a2a79614-aba4-44dd-928e-ccb32df6e0d0\") " pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.139041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-device-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-registration-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-system-cni-dir\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-var-lib-kubelet\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-run-multus-certs\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acb7ae1c-13d5-4745-aa4d-8e5f951e8f15-host\") pod \"node-ca-zh85g\" (UID: \"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15\") " pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-sys\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/384ea14e-b6d2-4db5-8b15-d3990b168f06-tmp\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7f9ed7d-0ea5-4c26-b542-703da8f84edb-hosts-file\") pod \"node-resolver-9pgsd\" (UID: \"b7f9ed7d-0ea5-4c26-b542-703da8f84edb\") " pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b181e08-247f-4e86-bf19-6a71d5d9edeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70f46be3-8082-4906-8168-ca1dff840646-agent-certs\") pod \"konnectivity-agent-zhnkf\" (UID: \"70f46be3-8082-4906-8168-ca1dff840646\") " pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-kubelet\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-run-netns\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-run-openvswitch\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.140057 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-run-multus-certs\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e943c3d7-af76-4889-9ab2-a272b6aef96a-ovnkube-config\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ce0de65-1777-4f99-ae24-df43d786f881-kubelet-config\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-sysconfig\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f9ed7d-0ea5-4c26-b542-703da8f84edb-tmp-dir\") pod \"node-resolver-9pgsd\" (UID: \"b7f9ed7d-0ea5-4c26-b542-703da8f84edb\") " pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhjl\" (UniqueName: \"kubernetes.io/projected/a2a79614-aba4-44dd-928e-ccb32df6e0d0-kube-api-access-gvhjl\") pod \"iptables-alerter-k2xhd\" (UID: \"a2a79614-aba4-44dd-928e-ccb32df6e0d0\") " pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b181e08-247f-4e86-bf19-6a71d5d9edeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc6a0944-4d3b-49ad-9f8f-6b438e040378-cni-binary-copy\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2a79614-aba4-44dd-928e-ccb32df6e0d0-iptables-alerter-script\") pod \"iptables-alerter-k2xhd\" (UID: \"a2a79614-aba4-44dd-928e-ccb32df6e0d0\") " pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-var-lib-kubelet\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acb7ae1c-13d5-4745-aa4d-8e5f951e8f15-host\") pod \"node-ca-zh85g\" (UID: \"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15\") " pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-registration-dir\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-kubelet\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.138967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7f9ed7d-0ea5-4c26-b542-703da8f84edb-hosts-file\") pod \"node-resolver-9pgsd\" (UID: \"b7f9ed7d-0ea5-4c26-b542-703da8f84edb\") " pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-sys\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-run-netns\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.139044    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:54.140970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-sysconfig\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.139107    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs podName:dd44d99b-8aab-441b-a376-af9a0d55f130 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:54.639084792 +0000 UTC m=+2.086179306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs") pod "network-metrics-daemon-6fpgx" (UID: "dd44d99b-8aab-441b-a376-af9a0d55f130") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8ce0de65-1777-4f99-ae24-df43d786f881-kubelet-config\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48993031-1a2a-4648-a040-79f0c8c09a1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-system-cni-dir\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwkn\" (UniqueName: \"kubernetes.io/projected/48993031-1a2a-4648-a040-79f0c8c09a1e-kube-api-access-6nwkn\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-run-ovn\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-kubernetes\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70f46be3-8082-4906-8168-ca1dff840646-konnectivity-ca\") pod \"konnectivity-agent-zhnkf\" (UID: \"70f46be3-8082-4906-8168-ca1dff840646\") " pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b181e08-247f-4e86-bf19-6a71d5d9edeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f9ed7d-0ea5-4c26-b542-703da8f84edb-tmp-dir\") pod \"node-resolver-9pgsd\" (UID: \"b7f9ed7d-0ea5-4c26-b542-703da8f84edb\") " pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-systemd\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-systemd\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b181e08-247f-4e86-bf19-6a71d5d9edeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-conf-dir\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.141680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b181e08-247f-4e86-bf19-6a71d5d9edeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ce0de65-1777-4f99-ae24-df43d786f881-dbus\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-socket-dir-parent\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-hostroot\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-etc-kubernetes\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-log-socket\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-run-openvswitch\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-run-netns\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-run-ovn\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-cni-bin\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-cni-netd\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-daemon-config\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b181e08-247f-4e86-bf19-6a71d5d9edeb-cnibin\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b181e08-247f-4e86-bf19-6a71d5d9edeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-var-lib-openvswitch\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-etc-kubernetes\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-log-socket\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-run-netns\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-cni-bin\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-cni-netd\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/70f46be3-8082-4906-8168-ca1dff840646-konnectivity-ca\") pod \"konnectivity-agent-zhnkf\" (UID: \"70f46be3-8082-4906-8168-ca1dff840646\") " pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.140387    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.140446    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret podName:8ce0de65-1777-4f99-ae24-df43d786f881 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:54.640428567 +0000 UTC m=+2.087523079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret") pod "global-pull-secret-syncer-rfcc8" (UID: "8ce0de65-1777-4f99-ae24-df43d786f881") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e943c3d7-af76-4889-9ab2-a272b6aef96a-ovnkube-script-lib\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-kubernetes\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8ce0de65-1777-4f99-ae24-df43d786f881-dbus\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-conf-dir\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-socket-dir-parent\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-hostroot\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.139987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e943c3d7-af76-4889-9ab2-a272b6aef96a-ovnkube-script-lib\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc6a0944-4d3b-49ad-9f8f-6b438e040378-multus-daemon-config\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.142990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-run-k8s-cni-cncf-io\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5sqf\" (UniqueName: \"kubernetes.io/projected/fc6a0944-4d3b-49ad-9f8f-6b438e040378-kube-api-access-x5sqf\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b181e08-247f-4e86-bf19-6a71d5d9edeb-system-cni-dir\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc6a0944-4d3b-49ad-9f8f-6b438e040378-host-run-k8s-cni-cncf-io\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-modprobe-d\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.140928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-modprobe-d\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.141031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b181e08-247f-4e86-bf19-6a71d5d9edeb-cnibin\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.141067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b181e08-247f-4e86-bf19-6a71d5d9edeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.141081    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e943c3d7-af76-4889-9ab2-a272b6aef96a-var-lib-openvswitch\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.141125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b181e08-247f-4e86-bf19-6a71d5d9edeb-system-cni-dir\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.141242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e943c3d7-af76-4889-9ab2-a272b6aef96a-ovnkube-config\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.142559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/384ea14e-b6d2-4db5-8b15-d3990b168f06-etc-tuned\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.142626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/384ea14e-b6d2-4db5-8b15-d3990b168f06-tmp\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.142805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/70f46be3-8082-4906-8168-ca1dff840646-agent-certs\") pod \"konnectivity-agent-zhnkf\" (UID: \"70f46be3-8082-4906-8168-ca1dff840646\") " pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:44:54.143621 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.143016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e943c3d7-af76-4889-9ab2-a272b6aef96a-ovn-node-metrics-cert\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.157428 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.157404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8r8\" (UniqueName: \"kubernetes.io/projected/dd44d99b-8aab-441b-a376-af9a0d55f130-kube-api-access-rb8r8\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:54.158703 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.158686    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:54.158703 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.158704    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:54.158815 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.158715    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xlwwm for pod openshift-network-diagnostics/network-check-target-n5x8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:54.158815 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.158769    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm podName:71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:54.65875666 +0000 UTC m=+2.105851170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xlwwm" (UniqueName: "kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm") pod "network-check-target-n5x8n" (UID: "71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:54.163372 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.163342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5sqf\" (UniqueName: \"kubernetes.io/projected/fc6a0944-4d3b-49ad-9f8f-6b438e040378-kube-api-access-x5sqf\") pod \"multus-srgnm\" (UID: \"fc6a0944-4d3b-49ad-9f8f-6b438e040378\") " pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.163532 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.163389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjc9\" (UniqueName: \"kubernetes.io/projected/acb7ae1c-13d5-4745-aa4d-8e5f951e8f15-kube-api-access-vpjc9\") pod \"node-ca-zh85g\" (UID: \"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15\") " pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.163532 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.163463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc784\" (UniqueName: \"kubernetes.io/projected/e943c3d7-af76-4889-9ab2-a272b6aef96a-kube-api-access-vc784\") pod \"ovnkube-node-knlxg\" (UID: \"e943c3d7-af76-4889-9ab2-a272b6aef96a\") " pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.163864 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.163838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7cpk\" (UniqueName: \"kubernetes.io/projected/1b181e08-247f-4e86-bf19-6a71d5d9edeb-kube-api-access-b7cpk\") pod \"multus-additional-cni-plugins-r2949\" (UID: \"1b181e08-247f-4e86-bf19-6a71d5d9edeb\") " pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.164205 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.164180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwkn\" (UniqueName: \"kubernetes.io/projected/48993031-1a2a-4648-a040-79f0c8c09a1e-kube-api-access-6nwkn\") pod \"aws-ebs-csi-driver-node-pkxmn\" (UID: \"48993031-1a2a-4648-a040-79f0c8c09a1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.164287 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.164242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ws5\" (UniqueName: \"kubernetes.io/projected/384ea14e-b6d2-4db5-8b15-d3990b168f06-kube-api-access-f5ws5\") pod \"tuned-8whgg\" (UID: \"384ea14e-b6d2-4db5-8b15-d3990b168f06\") " pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.164432 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.164412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhjl\" (UniqueName: \"kubernetes.io/projected/a2a79614-aba4-44dd-928e-ccb32df6e0d0-kube-api-access-gvhjl\") pod \"iptables-alerter-k2xhd\" (UID: \"a2a79614-aba4-44dd-928e-ccb32df6e0d0\") " pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.165083 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.165066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5pd\" (UniqueName: \"kubernetes.io/projected/b7f9ed7d-0ea5-4c26-b542-703da8f84edb-kube-api-access-2g5pd\") pod \"node-resolver-9pgsd\" (UID: \"b7f9ed7d-0ea5-4c26-b542-703da8f84edb\") " pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.185728 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.185703    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:54.215228 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.215161    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae49c932e0b671a70c1b9b1effab98ee.slice/crio-649b083f25c56819a88da13a8597417c3585ad54cfb1f0f8f8655b25e74f956c WatchSource:0}: Error finding container 649b083f25c56819a88da13a8597417c3585ad54cfb1f0f8f8655b25e74f956c: Status 404 returned error can't find the container with id 649b083f25c56819a88da13a8597417c3585ad54cfb1f0f8f8655b25e74f956c
Sep 16 18:44:54.215494 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.215471    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240b210d213d06d9b2a5c1727b8ee1a0.slice/crio-0777d900b8c33a2ef313bb3fe966bf62ae2f4bbce593ba0f3c5782fd007b376d WatchSource:0}: Error finding container 0777d900b8c33a2ef313bb3fe966bf62ae2f4bbce593ba0f3c5782fd007b376d: Status 404 returned error can't find the container with id 0777d900b8c33a2ef313bb3fe966bf62ae2f4bbce593ba0f3c5782fd007b376d
Sep 16 18:44:54.221242 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.221219    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:44:54.273009 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.272971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8whgg"
Sep 16 18:44:54.284001 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.283977    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384ea14e_b6d2_4db5_8b15_d3990b168f06.slice/crio-0b105667f5db8e9d9e105e44f08cdb100bbb133e82e2814725f8309bb81095ce WatchSource:0}: Error finding container 0b105667f5db8e9d9e105e44f08cdb100bbb133e82e2814725f8309bb81095ce: Status 404 returned error can't find the container with id 0b105667f5db8e9d9e105e44f08cdb100bbb133e82e2814725f8309bb81095ce
Sep 16 18:44:54.288275 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.288257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9pgsd"
Sep 16 18:44:54.296913 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.296879    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f9ed7d_0ea5_4c26_b542_703da8f84edb.slice/crio-e935db5bb2ef6a3ff851c1965489737889e4bb867e80cb73ea462273055b3dc4 WatchSource:0}: Error finding container e935db5bb2ef6a3ff851c1965489737889e4bb867e80cb73ea462273055b3dc4: Status 404 returned error can't find the container with id e935db5bb2ef6a3ff851c1965489737889e4bb867e80cb73ea462273055b3dc4
Sep 16 18:44:54.302188 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.302159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zh85g"
Sep 16 18:44:54.310998 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.310974    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb7ae1c_13d5_4745_aa4d_8e5f951e8f15.slice/crio-88d49470d7e74c5ac02084a6c8f3d876b5777af7bbe9aea86ed1c5738bb21f9d WatchSource:0}: Error finding container 88d49470d7e74c5ac02084a6c8f3d876b5777af7bbe9aea86ed1c5738bb21f9d: Status 404 returned error can't find the container with id 88d49470d7e74c5ac02084a6c8f3d876b5777af7bbe9aea86ed1c5738bb21f9d
Sep 16 18:44:54.319564 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.319535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2949"
Sep 16 18:44:54.328350 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.328328    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b181e08_247f_4e86_bf19_6a71d5d9edeb.slice/crio-7adfe6a3253e3d992606a47f9115da3e39a6178fd6021f0178e2a2cb94a3eec9 WatchSource:0}: Error finding container 7adfe6a3253e3d992606a47f9115da3e39a6178fd6021f0178e2a2cb94a3eec9: Status 404 returned error can't find the container with id 7adfe6a3253e3d992606a47f9115da3e39a6178fd6021f0178e2a2cb94a3eec9
Sep 16 18:44:54.334565 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.334531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-srgnm"
Sep 16 18:44:54.342696 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.342672    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6a0944_4d3b_49ad_9f8f_6b438e040378.slice/crio-fd680569df0a990bda6dbd001fd80ccd8313ef60509553e84a2eb4716040d752 WatchSource:0}: Error finding container fd680569df0a990bda6dbd001fd80ccd8313ef60509553e84a2eb4716040d752: Status 404 returned error can't find the container with id fd680569df0a990bda6dbd001fd80ccd8313ef60509553e84a2eb4716040d752
Sep 16 18:44:54.355198 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.355171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:44:54.363628 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.363600    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f46be3_8082_4906_8168_ca1dff840646.slice/crio-a19e1631ced140c9cb266e9f6799086741d34e642e61a0336680b8c1459b714d WatchSource:0}: Error finding container a19e1631ced140c9cb266e9f6799086741d34e642e61a0336680b8c1459b714d: Status 404 returned error can't find the container with id a19e1631ced140c9cb266e9f6799086741d34e642e61a0336680b8c1459b714d
Sep 16 18:44:54.373065 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.373043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn"
Sep 16 18:44:54.381956 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.381931    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48993031_1a2a_4648_a040_79f0c8c09a1e.slice/crio-b66cf558c9c759189476c6f0c01ccedf670e895d2695f6a5cf265c5044d9c30f WatchSource:0}: Error finding container b66cf558c9c759189476c6f0c01ccedf670e895d2695f6a5cf265c5044d9c30f: Status 404 returned error can't find the container with id b66cf558c9c759189476c6f0c01ccedf670e895d2695f6a5cf265c5044d9c30f
Sep 16 18:44:54.386577 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.386561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k2xhd"
Sep 16 18:44:54.391744 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.391724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:44:54.396471 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.396448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a79614_aba4_44dd_928e_ccb32df6e0d0.slice/crio-cf2c8c4860c29cf0ebd0f5551d260fc608311999a79bd1ba955fff007df676a4 WatchSource:0}: Error finding container cf2c8c4860c29cf0ebd0f5551d260fc608311999a79bd1ba955fff007df676a4: Status 404 returned error can't find the container with id cf2c8c4860c29cf0ebd0f5551d260fc608311999a79bd1ba955fff007df676a4
Sep 16 18:44:54.401256 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:44:54.401234    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode943c3d7_af76_4889_9ab2_a272b6aef96a.slice/crio-21df941ad2ebcb9fdba22895b1edd518cb22c3e31b74d3507778b975c423a125 WatchSource:0}: Error finding container 21df941ad2ebcb9fdba22895b1edd518cb22c3e31b74d3507778b975c423a125: Status 404 returned error can't find the container with id 21df941ad2ebcb9fdba22895b1edd518cb22c3e31b74d3507778b975c423a125
Sep 16 18:44:54.403833 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.403814    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-knlxg_openshift-ovn-kubernetes(e943c3d7-af76-4889-9ab2-a272b6aef96a): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 16 18:44:54.405494 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.405476    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-knlxg_openshift-ovn-kubernetes(e943c3d7-af76-4889-9ab2-a272b6aef96a): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 16 18:44:54.408901 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.408836    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" podUID="e943c3d7-af76-4889-9ab2-a272b6aef96a"
Sep 16 18:44:54.645400 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.645294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:54.645400 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.645345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:54.645651 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.645467    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:54.645651 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.645465    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:54.645651 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.645525    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret podName:8ce0de65-1777-4f99-ae24-df43d786f881 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:55.645511062 +0000 UTC m=+3.092605571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret") pod "global-pull-secret-syncer-rfcc8" (UID: "8ce0de65-1777-4f99-ae24-df43d786f881") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:54.645651 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.645601    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs podName:dd44d99b-8aab-441b-a376-af9a0d55f130 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:55.645581741 +0000 UTC m=+3.092676265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs") pod "network-metrics-daemon-6fpgx" (UID: "dd44d99b-8aab-441b-a376-af9a0d55f130") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:54.746398 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:54.745962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:44:54.746398 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.746117    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:54.746398 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.746137    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:54.746398 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.746147    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xlwwm for pod openshift-network-diagnostics/network-check-target-n5x8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:54.746398 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:54.746214    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm podName:71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:55.746192741 +0000 UTC m=+3.193287274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlwwm" (UniqueName: "kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm") pod "network-check-target-n5x8n" (UID: "71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:55.023744 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.023478    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:54 +0000 UTC" deadline="2028-05-08 20:14:09.021976475 +0000 UTC"
Sep 16 18:44:55.023744 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.023610    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14401h29m13.998371977s"
Sep 16 18:44:55.080972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.080906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerStarted","Data":"21df941ad2ebcb9fdba22895b1edd518cb22c3e31b74d3507778b975c423a125"}
Sep 16 18:44:55.115619 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.115483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k2xhd" event={"ID":"a2a79614-aba4-44dd-928e-ccb32df6e0d0","Type":"ContainerStarted","Data":"cf2c8c4860c29cf0ebd0f5551d260fc608311999a79bd1ba955fff007df676a4"}
Sep 16 18:44:55.123954 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.123923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zhnkf" event={"ID":"70f46be3-8082-4906-8168-ca1dff840646","Type":"ContainerStarted","Data":"a19e1631ced140c9cb266e9f6799086741d34e642e61a0336680b8c1459b714d"}
Sep 16 18:44:55.128812 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:55.128400    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" podUID="e943c3d7-af76-4889-9ab2-a272b6aef96a"
Sep 16 18:44:55.128812 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.128694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-srgnm" event={"ID":"fc6a0944-4d3b-49ad-9f8f-6b438e040378","Type":"ContainerStarted","Data":"fd680569df0a990bda6dbd001fd80ccd8313ef60509553e84a2eb4716040d752"}
Sep 16 18:44:55.131886 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.131858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2949" event={"ID":"1b181e08-247f-4e86-bf19-6a71d5d9edeb","Type":"ContainerStarted","Data":"7adfe6a3253e3d992606a47f9115da3e39a6178fd6021f0178e2a2cb94a3eec9"}
Sep 16 18:44:55.137832 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.137771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8whgg" event={"ID":"384ea14e-b6d2-4db5-8b15-d3990b168f06","Type":"ContainerStarted","Data":"0b105667f5db8e9d9e105e44f08cdb100bbb133e82e2814725f8309bb81095ce"}
Sep 16 18:44:55.151369 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.151290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal" event={"ID":"ae49c932e0b671a70c1b9b1effab98ee","Type":"ContainerStarted","Data":"649b083f25c56819a88da13a8597417c3585ad54cfb1f0f8f8655b25e74f956c"}
Sep 16 18:44:55.167286 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.167247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn" event={"ID":"48993031-1a2a-4648-a040-79f0c8c09a1e","Type":"ContainerStarted","Data":"b66cf558c9c759189476c6f0c01ccedf670e895d2695f6a5cf265c5044d9c30f"}
Sep 16 18:44:55.177720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.177684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zh85g" event={"ID":"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15","Type":"ContainerStarted","Data":"88d49470d7e74c5ac02084a6c8f3d876b5777af7bbe9aea86ed1c5738bb21f9d"}
Sep 16 18:44:55.188077 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.187797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9pgsd" event={"ID":"b7f9ed7d-0ea5-4c26-b542-703da8f84edb","Type":"ContainerStarted","Data":"e935db5bb2ef6a3ff851c1965489737889e4bb867e80cb73ea462273055b3dc4"}
Sep 16 18:44:55.196572 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.196504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal" event={"ID":"240b210d213d06d9b2a5c1727b8ee1a0","Type":"ContainerStarted","Data":"0777d900b8c33a2ef313bb3fe966bf62ae2f4bbce593ba0f3c5782fd007b376d"}
Sep 16 18:44:55.390244 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.390151    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:55.653951 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.653857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:55.653951 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.653916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:55.654184 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:55.654047    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:55.654184 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:55.654109    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret podName:8ce0de65-1777-4f99-ae24-df43d786f881 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:57.654091856 +0000 UTC m=+5.101186370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret") pod "global-pull-secret-syncer-rfcc8" (UID: "8ce0de65-1777-4f99-ae24-df43d786f881") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:55.654539 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:55.654515    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:55.654653 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:55.654565    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs podName:dd44d99b-8aab-441b-a376-af9a0d55f130 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:57.654550599 +0000 UTC m=+5.101645112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs") pod "network-metrics-daemon-6fpgx" (UID: "dd44d99b-8aab-441b-a376-af9a0d55f130") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:55.755431 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:55.754734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:44:55.755431 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:55.754943    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:55.755431 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:55.754968    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:55.755431 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:55.754983    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xlwwm for pod openshift-network-diagnostics/network-check-target-n5x8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:55.755431 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:55.755090    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm podName:71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:57.755027953 +0000 UTC m=+5.202122466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlwwm" (UniqueName: "kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm") pod "network-check-target-n5x8n" (UID: "71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:56.024251 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:56.024140    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:54 +0000 UTC" deadline="2028-05-27 14:19:46.069293802 +0000 UTC"
Sep 16 18:44:56.024251 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:56.024180    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14851h34m50.045119063s"
Sep 16 18:44:56.057686 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:56.057650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:56.057930 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:56.057788    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:44:56.058202 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:56.058180    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:44:56.058302 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:56.058282    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:44:56.058370 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:56.058355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:56.058462 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:56.058446    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:44:56.647632 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:56.647381    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:57.673991 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:57.673244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:57.673991 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:57.673375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:57.673991 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:57.673525    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:57.673991 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:57.673609    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs podName:dd44d99b-8aab-441b-a376-af9a0d55f130 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:01.673587703 +0000 UTC m=+9.120682213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs") pod "network-metrics-daemon-6fpgx" (UID: "dd44d99b-8aab-441b-a376-af9a0d55f130") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:57.673991 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:57.673730    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:57.673991 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:57.673781    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret podName:8ce0de65-1777-4f99-ae24-df43d786f881 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:01.673766951 +0000 UTC m=+9.120861473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret") pod "global-pull-secret-syncer-rfcc8" (UID: "8ce0de65-1777-4f99-ae24-df43d786f881") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:57.773947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:57.773906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:44:57.774124 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:57.774075    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:57.774124 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:57.774093    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:57.774124 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:57.774106    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xlwwm for pod openshift-network-diagnostics/network-check-target-n5x8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:57.774293 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:57.774166    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm podName:71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:01.774148509 +0000 UTC m=+9.221243033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlwwm" (UniqueName: "kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm") pod "network-check-target-n5x8n" (UID: "71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:58.058171 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:58.058132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:44:58.058338 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:58.058184    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:44:58.058338 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:44:58.058145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:44:58.058338 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:58.058313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:44:58.058541 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:58.058499    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:44:58.058615 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:44:58.058589    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:00.057723 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:00.057688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:00.058201 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:00.057725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:00.058201 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:00.057728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:00.058201 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:00.057820    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:00.058201 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:00.057928    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:00.058201 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:00.058022    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:01.704347 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:01.704306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:01.704971 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:01.704363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:01.704971 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:01.704503    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:01.704971 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:01.704563    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret podName:8ce0de65-1777-4f99-ae24-df43d786f881 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.704546623 +0000 UTC m=+17.151641132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret") pod "global-pull-secret-syncer-rfcc8" (UID: "8ce0de65-1777-4f99-ae24-df43d786f881") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:01.704971 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:01.704501    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:01.704971 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:01.704652    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs podName:dd44d99b-8aab-441b-a376-af9a0d55f130 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.70463409 +0000 UTC m=+17.151728605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs") pod "network-metrics-daemon-6fpgx" (UID: "dd44d99b-8aab-441b-a376-af9a0d55f130") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:01.805292 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:01.805202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:01.805458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:01.805355    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:01.805458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:01.805372    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:01.805458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:01.805381    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xlwwm for pod openshift-network-diagnostics/network-check-target-n5x8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:01.805458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:01.805441    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm podName:71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.805426466 +0000 UTC m=+17.252520973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlwwm" (UniqueName: "kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm") pod "network-check-target-n5x8n" (UID: "71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:02.057815 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:02.057633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:02.057815 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:02.057670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:02.057815 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:02.057670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:02.057815 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:02.057775    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:02.058113 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:02.057874    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:02.058113 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:02.057961    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:04.057696 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:04.057611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:04.057696 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:04.057635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:04.057696 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:04.057635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:04.058193 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:04.057748    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:04.058193 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:04.057889    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:04.058193 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:04.057961    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:06.058018 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:06.057982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:06.058465 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:06.057996    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:06.058465 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:06.057997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:06.058465 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:06.058121    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:06.058465 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:06.058229    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:06.058465 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:06.058326    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:08.058515 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:08.058243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:08.059097 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:08.058398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:08.059097 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:08.058626    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:08.059097 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:08.058434    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:08.059097 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:08.058728    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:08.059097 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:08.058847    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:08.225492 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:08.225350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-srgnm" event={"ID":"fc6a0944-4d3b-49ad-9f8f-6b438e040378","Type":"ContainerStarted","Data":"ce2e4ddd32d317414898cd62dbdf4a41a3591d4f6195b772c97c89f48095256b"}
Sep 16 18:45:08.228419 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:08.228387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8whgg" event={"ID":"384ea14e-b6d2-4db5-8b15-d3990b168f06","Type":"ContainerStarted","Data":"7722c70ff3a79e7d9df2085ff2941541ff5b333ddd182e0d0530b7003050073a"}
Sep 16 18:45:08.230733 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:08.230705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal" event={"ID":"240b210d213d06d9b2a5c1727b8ee1a0","Type":"ContainerStarted","Data":"57906998cd9da3ebf90ec8afde79c5340d95bd45c96624baccac3275e1e7ca91"}
Sep 16 18:45:08.245319 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:08.245257    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-srgnm" podStartSLOduration=1.804190927 podStartE2EDuration="15.245236311s" totalImagesPullingTime="13.441045384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.244882198 +0000 UTC m=+15.691976749" watchObservedRunningTime="2026-09-16 18:45:08.245236311 +0000 UTC m=+15.692330843"
Sep 16 18:45:08.284339 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:08.284284    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-161.ec2.internal" podStartSLOduration=15.28426284 podStartE2EDuration="15.28426284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.262636466 +0000 UTC m=+15.709731014" watchObservedRunningTime="2026-09-16 18:45:08.28426284 +0000 UTC m=+15.731357372"
Sep 16 18:45:08.284538 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:08.284375    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8whgg" podStartSLOduration=1.747736075 podStartE2EDuration="15.284366867s" totalImagesPullingTime="13.536630792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.283919175 +0000 UTC m=+15.731013734" watchObservedRunningTime="2026-09-16 18:45:08.284366867 +0000 UTC m=+15.731461400"
Sep 16 18:45:09.234762 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.234721    2578 generic.go:356] "Generic (PLEG): container finished" podID="ae49c932e0b671a70c1b9b1effab98ee" containerID="c171aae659d2b6ff8554a5c9745b7a03898e1deb1aabb34b50d70690ef92a563" exitCode=0
Sep 16 18:45:09.235236 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.234814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal" event={"ID":"ae49c932e0b671a70c1b9b1effab98ee","Type":"ContainerDied","Data":"c171aae659d2b6ff8554a5c9745b7a03898e1deb1aabb34b50d70690ef92a563"}
Sep 16 18:45:09.236785 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.236756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn" event={"ID":"48993031-1a2a-4648-a040-79f0c8c09a1e","Type":"ContainerStarted","Data":"841a4f05527be1bac44d7abff25a65b605ce8b4f76178b0bf909b941407769ad"}
Sep 16 18:45:09.238707 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.238551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zh85g" event={"ID":"acb7ae1c-13d5-4745-aa4d-8e5f951e8f15","Type":"ContainerStarted","Data":"8f612beafce009fb2b52909cc2540c3e1c9cc0aaf2f58b7137c9ce37e1c8cb9b"}
Sep 16 18:45:09.240978 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.240953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9pgsd" event={"ID":"b7f9ed7d-0ea5-4c26-b542-703da8f84edb","Type":"ContainerStarted","Data":"6db578ee4a0ce01e5e3b25725cf17d56565e05bbe2145477790d33c88b2f7fe9"}
Sep 16 18:45:09.242899 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.242877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zhnkf" event={"ID":"70f46be3-8082-4906-8168-ca1dff840646","Type":"ContainerStarted","Data":"ddea7b276404a45423d14f0bfa2478e4384265d68327643af0ce01af88d4fcd6"}
Sep 16 18:45:09.244617 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.244591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2949" event={"ID":"1b181e08-247f-4e86-bf19-6a71d5d9edeb","Type":"ContainerStarted","Data":"d4e4bc257158001f127e9e7f799530fba70de22ec307dae417805249b80a944e"}
Sep 16 18:45:09.301063 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.301012    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zhnkf" podStartSLOduration=2.9647524 podStartE2EDuration="16.30098766s" totalImagesPullingTime="13.33623526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:09.280301649 +0000 UTC m=+16.727396215" watchObservedRunningTime="2026-09-16 18:45:09.30098766 +0000 UTC m=+16.748082192"
Sep 16 18:45:09.332063 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.332010    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9pgsd" podStartSLOduration=2.929542461 podStartE2EDuration="16.331991996s" totalImagesPullingTime="13.402449535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:09.301539399 +0000 UTC m=+16.748633944" watchObservedRunningTime="2026-09-16 18:45:09.331991996 +0000 UTC m=+16.779086526"
Sep 16 18:45:09.332260 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.332099    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zh85g" podStartSLOduration=6.989292097 podStartE2EDuration="16.332093741s" totalImagesPullingTime="9.342801644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:09.331617204 +0000 UTC m=+16.778711757" watchObservedRunningTime="2026-09-16 18:45:09.332093741 +0000 UTC m=+16.779188271"
Sep 16 18:45:09.769838 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.769804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:09.769944 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:09.769856    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:09.769944 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.769873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:09.769944 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:09.769922    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs podName:dd44d99b-8aab-441b-a376-af9a0d55f130 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:25.76990279 +0000 UTC m=+33.216997321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs") pod "network-metrics-daemon-6fpgx" (UID: "dd44d99b-8aab-441b-a376-af9a0d55f130") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:09.770066 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:09.769958    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:09.770066 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:09.769993    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret podName:8ce0de65-1777-4f99-ae24-df43d786f881 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:25.769982964 +0000 UTC m=+33.217077486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret") pod "global-pull-secret-syncer-rfcc8" (UID: "8ce0de65-1777-4f99-ae24-df43d786f881") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:09.870635 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:09.870594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:09.870811 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:09.870769    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:09.870811 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:09.870792    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:09.870811 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:09.870805    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xlwwm for pod openshift-network-diagnostics/network-check-target-n5x8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:09.870966 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:09.870875    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm podName:71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:25.870852588 +0000 UTC m=+33.317947098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlwwm" (UniqueName: "kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm") pod "network-check-target-n5x8n" (UID: "71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:10.058419 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.058333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:10.058601 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.058354    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:10.058601 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:10.058460    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:10.058601 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.058518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:10.058742 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:10.058611    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:10.058742 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:10.058678    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:10.248496 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.248242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k2xhd" event={"ID":"a2a79614-aba4-44dd-928e-ccb32df6e0d0","Type":"ContainerStarted","Data":"33090873b2a5ea1836a06d8539ad91951cc7f5ede3bfde70e3a592b80a8c5fd1"}
Sep 16 18:45:10.250041 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.250010    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b181e08-247f-4e86-bf19-6a71d5d9edeb" containerID="d4e4bc257158001f127e9e7f799530fba70de22ec307dae417805249b80a944e" exitCode=0
Sep 16 18:45:10.250167 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.250099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2949" event={"ID":"1b181e08-247f-4e86-bf19-6a71d5d9edeb","Type":"ContainerDied","Data":"d4e4bc257158001f127e9e7f799530fba70de22ec307dae417805249b80a944e"}
Sep 16 18:45:10.252420 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.252396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal" event={"ID":"ae49c932e0b671a70c1b9b1effab98ee","Type":"ContainerStarted","Data":"475d19df8ad3c452ea8579a3df05999ccb3dc5b5e5d09e6d038a47ffec6c1b68"}
Sep 16 18:45:10.270610 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.270547    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k2xhd" podStartSLOduration=3.967459392 podStartE2EDuration="17.270526282s" totalImagesPullingTime="13.30306689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:10.269780273 +0000 UTC m=+17.716874830" watchObservedRunningTime="2026-09-16 18:45:10.270526282 +0000 UTC m=+17.717620813"
Sep 16 18:45:10.278242 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.278218    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:45:10.286321 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.286271    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-161.ec2.internal" podStartSLOduration=17.286253761 podStartE2EDuration="17.286253761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:10.285725415 +0000 UTC m=+17.732819959" watchObservedRunningTime="2026-09-16 18:45:10.286253761 +0000 UTC m=+17.733348292"
Sep 16 18:45:10.984772 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.984465    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:10.278238762Z","UUID":"357595ee-7144-49a9-982b-291f740d8e1a","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:45:10.986955 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.986463    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:45:10.986955 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:10.986514    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:45:11.258379 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:11.258160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn" event={"ID":"48993031-1a2a-4648-a040-79f0c8c09a1e","Type":"ContainerStarted","Data":"4f66314639ad336ae98e561cd0a70faa17c9e23e164c5446993839e6e3e75f28"}
Sep 16 18:45:11.258379 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:11.258277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn" event={"ID":"48993031-1a2a-4648-a040-79f0c8c09a1e","Type":"ContainerStarted","Data":"d9119919fa6ec8d35269078b20ecf3d37854b3a859c800f4c41a48af078c4221"}
Sep 16 18:45:11.278776 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:11.278725    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pkxmn" podStartSLOduration=2.728572246 podStartE2EDuration="18.278709017s" totalImagesPullingTime="15.550136771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:11.278643872 +0000 UTC m=+18.725738425" watchObservedRunningTime="2026-09-16 18:45:11.278709017 +0000 UTC m=+18.725803578"
Sep 16 18:45:12.057840 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:12.057695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:12.057840 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:12.057716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:12.057840 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:12.057837    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:12.058137 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:12.057890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:12.058137 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:12.057983    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:12.058137 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:12.058068    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:13.814101 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:13.814067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:45:13.814912 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:13.814770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:45:14.057661 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:14.057622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:14.057842 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:14.057639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:14.057842 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:14.057759    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:14.057842 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:14.057807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:14.057996 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:14.057885    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:14.058050 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:14.057985    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:16.058102 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:16.058064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:16.058592 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:16.058114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:16.058592 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:16.058064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:16.058592 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:16.058191    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:16.058592 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:16.058276    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:16.058592 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:16.058354    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:18.057502 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:18.057455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:18.058014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:18.057557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:18.058014 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:18.057564    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:18.058014 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:18.057674    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:18.058014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:18.057719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:18.058014 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:18.057801    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:19.279128 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.278952    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b181e08-247f-4e86-bf19-6a71d5d9edeb" containerID="ee1ad3befddafdedd852da76e96c5446e5b22a180c86e8539db2d74bbfe54254" exitCode=0
Sep 16 18:45:19.279574 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.279034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2949" event={"ID":"1b181e08-247f-4e86-bf19-6a71d5d9edeb","Type":"ContainerDied","Data":"ee1ad3befddafdedd852da76e96c5446e5b22a180c86e8539db2d74bbfe54254"}
Sep 16 18:45:19.285006 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.284989    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:45:19.285342 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.285324    2578 generic.go:356] "Generic (PLEG): container finished" podID="e943c3d7-af76-4889-9ab2-a272b6aef96a" containerID="d91c6ac591bee3998f546221b5d9b7dc8fe472525aba5685b29d1b578702827c" exitCode=1
Sep 16 18:45:19.285395 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.285356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerStarted","Data":"99b9f557724ddc10e2cd801246a1d86e8214bbec6eb4ede9ad8a3ec2c13ac17c"}
Sep 16 18:45:19.285425 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.285408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerStarted","Data":"9a24bab794b43b8006e5146097df15d0cb139c5dc23b2328a82dd381304eaec4"}
Sep 16 18:45:19.285453 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.285442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerStarted","Data":"c26e1b7df544ae7fbdb0b1387bae746b39927c28a31bbb37202c76975ad66a56"}
Sep 16 18:45:19.285500 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.285492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerStarted","Data":"a18d6f8442e06f96c7e79e6db9a2f3d25d35fe17467bd7cf093056fc618b24aa"}
Sep 16 18:45:19.285537 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.285521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerDied","Data":"d91c6ac591bee3998f546221b5d9b7dc8fe472525aba5685b29d1b578702827c"}
Sep 16 18:45:19.285570 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:19.285557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerStarted","Data":"2cc54f118a5126af1261d40146322bfdc9a9661a3c95930f5399ad4674df38fd"}
Sep 16 18:45:20.058610 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:20.058416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:20.058711 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:20.058437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:20.058747 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:20.058717    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:20.058792 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:20.058436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:20.058829 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:20.058754    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:20.058874 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:20.058841    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:20.289954 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:20.289867    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b181e08-247f-4e86-bf19-6a71d5d9edeb" containerID="48e1ff9056012ed2a395560cce4ab4f09b60d2cd4fb1396f0871d5b059964f8f" exitCode=0
Sep 16 18:45:20.289954 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:20.289920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2949" event={"ID":"1b181e08-247f-4e86-bf19-6a71d5d9edeb","Type":"ContainerDied","Data":"48e1ff9056012ed2a395560cce4ab4f09b60d2cd4fb1396f0871d5b059964f8f"}
Sep 16 18:45:21.295302 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:21.295225    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:45:21.295726 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:21.295666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerStarted","Data":"89a059a914674d08dc618fe8062d9b5e0a31f46a6edf34cc80fa7cca53f556ed"}
Sep 16 18:45:21.298009 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:21.297985    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b181e08-247f-4e86-bf19-6a71d5d9edeb" containerID="2cfdd60c8b2c996bfb4a03c16112170aeaa504ee1961759958f06635fd04ec8a" exitCode=0
Sep 16 18:45:21.298116 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:21.298026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2949" event={"ID":"1b181e08-247f-4e86-bf19-6a71d5d9edeb","Type":"ContainerDied","Data":"2cfdd60c8b2c996bfb4a03c16112170aeaa504ee1961759958f06635fd04ec8a"}
Sep 16 18:45:22.058086 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:22.058048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:22.058269 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:22.058182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:22.058269 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:22.058184    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:22.058269 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:22.058231    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:22.058428 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:22.058317    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:22.058428 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:22.058411    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:22.444309 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:22.444228    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:45:22.444764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:22.444367    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:45:22.444917 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:22.444895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zhnkf"
Sep 16 18:45:23.307429 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:23.307351    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:45:23.307959 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:23.307925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerStarted","Data":"e300fbdc5951a4d855d49b0910ff13825f2c46ab3f10e8fa81ff3ab66596fa2f"}
Sep 16 18:45:23.308413 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:23.308397    2578 scope.go:117] "RemoveContainer" containerID="d91c6ac591bee3998f546221b5d9b7dc8fe472525aba5685b29d1b578702827c"
Sep 16 18:45:24.057536 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.057500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:24.058362 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.057540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:24.058362 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.057543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:24.058362 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:24.057628    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:24.058362 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:24.057703    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:24.058362 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:24.057823    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:24.314926 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.314855    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:45:24.315450 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.315420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" event={"ID":"e943c3d7-af76-4889-9ab2-a272b6aef96a","Type":"ContainerStarted","Data":"88d76033c3c18c8944f8d27f4ebe874a408a70e6d0947e65dc186f6b0cef802b"}
Sep 16 18:45:24.315945 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.315925    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:45:24.316077 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.316020    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:45:24.316139 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.316099    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:45:24.333161 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.333126    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:45:24.333585 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.333560    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:45:24.346680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:24.346631    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg" podStartSLOduration=7.505326743 podStartE2EDuration="31.34661167s" totalImagesPullingTime="23.841284927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-09-16 18:45:24.345998489 +0000 UTC m=+31.793093020" watchObservedRunningTime="2026-09-16 18:45:24.34661167 +0000 UTC m=+31.793706200"
Sep 16 18:45:25.067467 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:25.067201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rfcc8"]
Sep 16 18:45:25.068022 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:25.067588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:25.068022 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.067696    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:25.068022 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:25.067883    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n5x8n"]
Sep 16 18:45:25.068022 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:25.067963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:25.068244 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.068046    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:25.069954 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:25.069925    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6fpgx"]
Sep 16 18:45:25.070074 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:25.070042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:25.070163 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.070138    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:25.787832 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:25.787807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:25.787969 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:25.787851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:25.787969 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.787940    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:25.788083 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.787994    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs podName:dd44d99b-8aab-441b-a376-af9a0d55f130 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:57.787977017 +0000 UTC m=+65.235071528 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs") pod "network-metrics-daemon-6fpgx" (UID: "dd44d99b-8aab-441b-a376-af9a0d55f130") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:25.788083 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.787943    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:25.788083 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.788076    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret podName:8ce0de65-1777-4f99-ae24-df43d786f881 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:57.788057712 +0000 UTC m=+65.235152235 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret") pod "global-pull-secret-syncer-rfcc8" (UID: "8ce0de65-1777-4f99-ae24-df43d786f881") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:25.888320 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:25.888289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:25.888517 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.888436    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:25.888517 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.888454    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:25.888517 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.888463    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xlwwm for pod openshift-network-diagnostics/network-check-target-n5x8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:25.888631 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:25.888523    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm podName:71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:57.888509006 +0000 UTC m=+65.335603514 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlwwm" (UniqueName: "kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm") pod "network-check-target-n5x8n" (UID: "71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:26.322820 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:26.322791    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b181e08-247f-4e86-bf19-6a71d5d9edeb" containerID="cf93ee68c8ba337365718409419b1b0d116740a8f82896178bbbc4b4589e3df7" exitCode=0
Sep 16 18:45:26.323249 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:26.322874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2949" event={"ID":"1b181e08-247f-4e86-bf19-6a71d5d9edeb","Type":"ContainerDied","Data":"cf93ee68c8ba337365718409419b1b0d116740a8f82896178bbbc4b4589e3df7"}
Sep 16 18:45:27.058123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:27.058084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:27.058263 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:27.058092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:27.058263 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:27.058214    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:27.058338 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:27.058094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:27.058338 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:27.058271    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:27.058412 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:27.058342    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:27.329573 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:27.329541    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b181e08-247f-4e86-bf19-6a71d5d9edeb" containerID="dc645dd30caebaef80b5b67bf284d7623239f088788af3b569d5cdcf8bc8ec1b" exitCode=0
Sep 16 18:45:27.329993 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:27.329594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2949" event={"ID":"1b181e08-247f-4e86-bf19-6a71d5d9edeb","Type":"ContainerDied","Data":"dc645dd30caebaef80b5b67bf284d7623239f088788af3b569d5cdcf8bc8ec1b"}
Sep 16 18:45:28.335200 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:28.335015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2949" event={"ID":"1b181e08-247f-4e86-bf19-6a71d5d9edeb","Type":"ContainerStarted","Data":"4c7888f9934e7438a96c5471eecf6bcc907e94d25343d514117441c0efedf99a"}
Sep 16 18:45:28.360007 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:28.359960    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r2949" podStartSLOduration=8.376607321 podStartE2EDuration="35.359944618s" totalImagesPullingTime="26.983337297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:28.359827027 +0000 UTC m=+35.806921568" watchObservedRunningTime="2026-09-16 18:45:28.359944618 +0000 UTC m=+35.807039147"
Sep 16 18:45:29.058313 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:29.058285    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:29.058313 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:29.058311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:29.058537 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:29.058289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:29.058537 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:29.058408    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:29.058537 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:29.058461    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:29.058692 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:29.058561    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:31.058468 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.058430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:31.058468 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.058466    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:31.058983 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.058430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:31.058983 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:31.058557    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5x8n" podUID="71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48"
Sep 16 18:45:31.058983 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:31.058645    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rfcc8" podUID="8ce0de65-1777-4f99-ae24-df43d786f881"
Sep 16 18:45:31.059468 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:31.059377    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fpgx" podUID="dd44d99b-8aab-441b-a376-af9a0d55f130"
Sep 16 18:45:31.267883 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.267797    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-161.ec2.internal" event="NodeReady"
Sep 16 18:45:31.268040 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.267946    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:45:31.323357 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.323325    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h7lbj"]
Sep 16 18:45:31.343371 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.343345    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-djtj7"]
Sep 16 18:45:31.343558 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.343464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.346235 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.346212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:45:31.346235 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.346232    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:45:31.346458 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.346266    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dwfv6\""
Sep 16 18:45:31.367486 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.367442    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h7lbj"]
Sep 16 18:45:31.367651 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.367501    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-djtj7"]
Sep 16 18:45:31.367651 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.367591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-djtj7"
Sep 16 18:45:31.370227 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.370207    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:45:31.370346 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.370280    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dgnzh\""
Sep 16 18:45:31.370789 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.370768    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:45:31.370868 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.370801    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:45:31.422137 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.422103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6727w\" (UniqueName: \"kubernetes.io/projected/0e084045-524d-49ab-9191-7a4c3430cf0c-kube-api-access-6727w\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.422137 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.422141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.422329 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.422174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e084045-524d-49ab-9191-7a4c3430cf0c-tmp-dir\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.422329 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.422254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e084045-524d-49ab-9191-7a4c3430cf0c-config-volume\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.523141 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.523048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6727w\" (UniqueName: \"kubernetes.io/projected/0e084045-524d-49ab-9191-7a4c3430cf0c-kube-api-access-6727w\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.523141 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.523093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.523366 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:31.523190    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:31.523366 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.523210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fe145cc-75f4-4d0e-844e-4822424e1071-cert\") pod \"ingress-canary-djtj7\" (UID: \"5fe145cc-75f4-4d0e-844e-4822424e1071\") " pod="openshift-ingress-canary/ingress-canary-djtj7"
Sep 16 18:45:31.523366 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.523242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzhz\" (UniqueName: \"kubernetes.io/projected/5fe145cc-75f4-4d0e-844e-4822424e1071-kube-api-access-kfzhz\") pod \"ingress-canary-djtj7\" (UID: \"5fe145cc-75f4-4d0e-844e-4822424e1071\") " pod="openshift-ingress-canary/ingress-canary-djtj7"
Sep 16 18:45:31.523366 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:31.523259    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls podName:0e084045-524d-49ab-9191-7a4c3430cf0c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:32.02323853 +0000 UTC m=+39.470333052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls") pod "dns-default-h7lbj" (UID: "0e084045-524d-49ab-9191-7a4c3430cf0c") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:31.523366 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.523330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e084045-524d-49ab-9191-7a4c3430cf0c-tmp-dir\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.523652 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.523377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e084045-524d-49ab-9191-7a4c3430cf0c-config-volume\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.523708 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.523681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e084045-524d-49ab-9191-7a4c3430cf0c-tmp-dir\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.523986 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.523966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e084045-524d-49ab-9191-7a4c3430cf0c-config-volume\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.534442 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.534423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6727w\" (UniqueName: \"kubernetes.io/projected/0e084045-524d-49ab-9191-7a4c3430cf0c-kube-api-access-6727w\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:31.624609 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.624569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fe145cc-75f4-4d0e-844e-4822424e1071-cert\") pod \"ingress-canary-djtj7\" (UID: \"5fe145cc-75f4-4d0e-844e-4822424e1071\") " pod="openshift-ingress-canary/ingress-canary-djtj7"
Sep 16 18:45:31.624609 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.624612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzhz\" (UniqueName: \"kubernetes.io/projected/5fe145cc-75f4-4d0e-844e-4822424e1071-kube-api-access-kfzhz\") pod \"ingress-canary-djtj7\" (UID: \"5fe145cc-75f4-4d0e-844e-4822424e1071\") " pod="openshift-ingress-canary/ingress-canary-djtj7"
Sep 16 18:45:31.627008 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.626987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fe145cc-75f4-4d0e-844e-4822424e1071-cert\") pod \"ingress-canary-djtj7\" (UID: \"5fe145cc-75f4-4d0e-844e-4822424e1071\") " pod="openshift-ingress-canary/ingress-canary-djtj7"
Sep 16 18:45:31.632560 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.632536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzhz\" (UniqueName: \"kubernetes.io/projected/5fe145cc-75f4-4d0e-844e-4822424e1071-kube-api-access-kfzhz\") pod \"ingress-canary-djtj7\" (UID: \"5fe145cc-75f4-4d0e-844e-4822424e1071\") " pod="openshift-ingress-canary/ingress-canary-djtj7"
Sep 16 18:45:31.675657 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.675619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-djtj7"
Sep 16 18:45:31.996025 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:31.995913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-djtj7"]
Sep 16 18:45:31.999619 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:45:31.999593    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe145cc_75f4_4d0e_844e_4822424e1071.slice/crio-15ea6106df0407406d2c053bc675d892eb72281d9e47361bd83e3ff5973fb771 WatchSource:0}: Error finding container 15ea6106df0407406d2c053bc675d892eb72281d9e47361bd83e3ff5973fb771: Status 404 returned error can't find the container with id 15ea6106df0407406d2c053bc675d892eb72281d9e47361bd83e3ff5973fb771
Sep 16 18:45:32.027278 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:32.027247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:32.027404 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:32.027373    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:32.027459 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:32.027425    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls podName:0e084045-524d-49ab-9191-7a4c3430cf0c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:33.027409819 +0000 UTC m=+40.474504328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls") pod "dns-default-h7lbj" (UID: "0e084045-524d-49ab-9191-7a4c3430cf0c") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:32.345403 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:32.345370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-djtj7" event={"ID":"5fe145cc-75f4-4d0e-844e-4822424e1071","Type":"ContainerStarted","Data":"15ea6106df0407406d2c053bc675d892eb72281d9e47361bd83e3ff5973fb771"}
Sep 16 18:45:33.033399 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.033358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:33.033586 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:33.033511    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:33.033648 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:33.033585    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls podName:0e084045-524d-49ab-9191-7a4c3430cf0c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.033563053 +0000 UTC m=+42.480657572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls") pod "dns-default-h7lbj" (UID: "0e084045-524d-49ab-9191-7a4c3430cf0c") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:33.062668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.059975    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:33.062668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.060473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:33.062668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.061118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:33.066946 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.066920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:33.066946 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.066951    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:33.067162 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.066977    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pq9cp\""
Sep 16 18:45:33.067162 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.066980    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:33.067162 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.067040    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:33.067162 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:33.067048    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrh8k\""
Sep 16 18:45:35.053104 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:35.052919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:35.053104 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:35.053072    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:35.053532 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:35.053161    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls podName:0e084045-524d-49ab-9191-7a4c3430cf0c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.053144935 +0000 UTC m=+46.500239443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls") pod "dns-default-h7lbj" (UID: "0e084045-524d-49ab-9191-7a4c3430cf0c") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:35.353067 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:35.353032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-djtj7" event={"ID":"5fe145cc-75f4-4d0e-844e-4822424e1071","Type":"ContainerStarted","Data":"87c074a29a35cb622e80a842eed3e82b1b97b41ba09bacbcf59aa2dbff6d025b"}
Sep 16 18:45:35.371187 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:35.371133    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-djtj7" podStartSLOduration=1.878782314 podStartE2EDuration="4.371118335s" totalImagesPullingTime="2.492336021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:35.369589774 +0000 UTC m=+42.816684303" watchObservedRunningTime="2026-09-16 18:45:35.371118335 +0000 UTC m=+42.818212884"
Sep 16 18:45:39.079983 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:39.079940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:39.080544 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:39.080141    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:39.080544 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:39.080223    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls podName:0e084045-524d-49ab-9191-7a4c3430cf0c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.08020226 +0000 UTC m=+54.527296772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls") pod "dns-default-h7lbj" (UID: "0e084045-524d-49ab-9191-7a4c3430cf0c") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:47.132695 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:47.132636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:45:47.133081 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:47.132761    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:47.133081 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:47.132822    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls podName:0e084045-524d-49ab-9191-7a4c3430cf0c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:03.132808569 +0000 UTC m=+70.579903076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls") pod "dns-default-h7lbj" (UID: "0e084045-524d-49ab-9191-7a4c3430cf0c") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:56.333814 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:56.333785    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knlxg"
Sep 16 18:45:57.806146 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.806109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:57.806559 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.806188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:45:57.809064 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.809044    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:57.809113 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.809079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:57.817315 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:57.817290    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:45:57.817432 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:45:57.817367    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs podName:dd44d99b-8aab-441b-a376-af9a0d55f130 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:01.817344856 +0000 UTC m=+129.264439368 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs") pod "network-metrics-daemon-6fpgx" (UID: "dd44d99b-8aab-441b-a376-af9a0d55f130") : secret "metrics-daemon-secret" not found
Sep 16 18:45:57.820341 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.820323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8ce0de65-1777-4f99-ae24-df43d786f881-original-pull-secret\") pod \"global-pull-secret-syncer-rfcc8\" (UID: \"8ce0de65-1777-4f99-ae24-df43d786f881\") " pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:57.906510 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.906464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:57.909504 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.909464    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:57.920459 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.920443    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:57.930371 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.930351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwwm\" (UniqueName: \"kubernetes.io/projected/71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48-kube-api-access-xlwwm\") pod \"network-check-target-n5x8n\" (UID: \"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48\") " pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:57.978397 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.978364    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrh8k\""
Sep 16 18:45:57.983612 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.983592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rfcc8"
Sep 16 18:45:57.985729 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:57.985713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:45:58.262530 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:58.262440    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rfcc8"]
Sep 16 18:45:58.265617 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:58.265592    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n5x8n"]
Sep 16 18:45:58.266516 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:45:58.266438    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce0de65_1777_4f99_ae24_df43d786f881.slice/crio-24181c53f17f21f345966ea9e453d294d914eabf531ef8d309b1b21f7170e0ce WatchSource:0}: Error finding container 24181c53f17f21f345966ea9e453d294d914eabf531ef8d309b1b21f7170e0ce: Status 404 returned error can't find the container with id 24181c53f17f21f345966ea9e453d294d914eabf531ef8d309b1b21f7170e0ce
Sep 16 18:45:58.268903 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:45:58.268881    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bd5010_b5ce_4f9f_9adf_fa76ef4a5b48.slice/crio-eacb65c7af0e939141265324669d1ff09520470a02fe2b5b326cddaa061860fd WatchSource:0}: Error finding container eacb65c7af0e939141265324669d1ff09520470a02fe2b5b326cddaa061860fd: Status 404 returned error can't find the container with id eacb65c7af0e939141265324669d1ff09520470a02fe2b5b326cddaa061860fd
Sep 16 18:45:58.406632 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:58.406601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rfcc8" event={"ID":"8ce0de65-1777-4f99-ae24-df43d786f881","Type":"ContainerStarted","Data":"24181c53f17f21f345966ea9e453d294d914eabf531ef8d309b1b21f7170e0ce"}
Sep 16 18:45:58.407670 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:45:58.407647    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n5x8n" event={"ID":"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48","Type":"ContainerStarted","Data":"eacb65c7af0e939141265324669d1ff09520470a02fe2b5b326cddaa061860fd"}
Sep 16 18:46:03.141449 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:03.141405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:46:03.141874 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:03.141576    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:03.141874 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:03.141650    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls podName:0e084045-524d-49ab-9191-7a4c3430cf0c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:35.141624227 +0000 UTC m=+102.588718740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls") pod "dns-default-h7lbj" (UID: "0e084045-524d-49ab-9191-7a4c3430cf0c") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:03.424498 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:03.424388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n5x8n" event={"ID":"71bd5010-b5ce-4f9f-9adf-fa76ef4a5b48","Type":"ContainerStarted","Data":"4dfddb5cf21be9d3381f70ddab9369609c6038b6d67d694121fd5dd527e2b0e1"}
Sep 16 18:46:03.424498 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:03.424468    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:46:03.425958 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:03.425932    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rfcc8" event={"ID":"8ce0de65-1777-4f99-ae24-df43d786f881","Type":"ContainerStarted","Data":"4cc8e73dcd2e798f975a265e00785ba6eb4336faca8713c55cd165e47e5df864"}
Sep 16 18:46:03.443762 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:03.443715    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n5x8n" podStartSLOduration=66.116698585 podStartE2EDuration="1m10.443699219s" totalImagesPullingTime="4.327000634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:03.442904059 +0000 UTC m=+70.889998700" watchObservedRunningTime="2026-09-16 18:46:03.443699219 +0000 UTC m=+70.890793780"
Sep 16 18:46:03.464183 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:03.464134    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rfcc8" podStartSLOduration=66.131559726 podStartE2EDuration="1m10.4641179s" totalImagesPullingTime="4.332558174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:03.463725696 +0000 UTC m=+70.910820239" watchObservedRunningTime="2026-09-16 18:46:03.4641179 +0000 UTC m=+70.911212430"
Sep 16 18:46:05.150137 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.150104    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf"]
Sep 16 18:46:05.153598 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.153581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf"
Sep 16 18:46:05.156265 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.156239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-67dp9\""
Sep 16 18:46:05.156378 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.156319    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:05.156378 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.156319    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:05.162464 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.162441    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf"]
Sep 16 18:46:05.256173 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.256138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcrwp\" (UniqueName: \"kubernetes.io/projected/9761c3ca-de69-4601-98a8-bd173480628c-kube-api-access-qcrwp\") pod \"volume-data-source-validator-86695b7cc7-xgjxf\" (UID: \"9761c3ca-de69-4601-98a8-bd173480628c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf"
Sep 16 18:46:05.258989 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.258965    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"]
Sep 16 18:46:05.261746 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.261730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.265136 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.265115    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-f6h98"]
Sep 16 18:46:05.265265 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.265249    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-l88fn\""
Sep 16 18:46:05.265333 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.265159    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:46:05.266551 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.266523    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:46:05.266680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.266564    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:05.266680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.266606    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:05.268591 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.268559    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7df8cfc95d-tzm6f"]
Sep 16 18:46:05.268744 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.268718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.271236 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.271215    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:46:05.271418 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.271403    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4rx7v\""
Sep 16 18:46:05.271736 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.271713    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:46:05.271838 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.271795    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:05.271838 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.271818    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:05.271922 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.271887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.274690 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.274670    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:46:05.274790 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.274697    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lb47v\""
Sep 16 18:46:05.274898 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.274884    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:46:05.275413 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.275400    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:46:05.275749 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.275732    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:46:05.276054 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.275963    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:46:05.276186 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.276169    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:46:05.278197 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.278178    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:46:05.286843 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.286819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"]
Sep 16 18:46:05.287507 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.287471    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-f6h98"]
Sep 16 18:46:05.288185 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.288170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7df8cfc95d-tzm6f"]
Sep 16 18:46:05.350780 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.350746    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-48rtn"]
Sep 16 18:46:05.357030 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcrwp\" (UniqueName: \"kubernetes.io/projected/9761c3ca-de69-4601-98a8-bd173480628c-kube-api-access-qcrwp\") pod \"volume-data-source-validator-86695b7cc7-xgjxf\" (UID: \"9761c3ca-de69-4601-98a8-bd173480628c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf"
Sep 16 18:46:05.357147 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-default-certificate\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.357147 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.357147 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d02c7f84-f3da-432a-9cc3-2588b5256190-config\") pod \"service-ca-operator-cf7c776b7-zgkp7\" (UID: \"d02c7f84-f3da-432a-9cc3-2588b5256190\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.357147 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-stats-auth\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.357281 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d02c7f84-f3da-432a-9cc3-2588b5256190-serving-cert\") pod \"service-ca-operator-cf7c776b7-zgkp7\" (UID: \"d02c7f84-f3da-432a-9cc3-2588b5256190\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.357281 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391ce6ca-1cf6-458a-a904-59997c7eb6db-trusted-ca\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.357281 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/391ce6ca-1cf6-458a-a904-59997c7eb6db-config\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.357281 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/391ce6ca-1cf6-458a-a904-59997c7eb6db-serving-cert\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.357281 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.357457 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxqkv\" (UniqueName: \"kubernetes.io/projected/d02c7f84-f3da-432a-9cc3-2588b5256190-kube-api-access-dxqkv\") pod \"service-ca-operator-cf7c776b7-zgkp7\" (UID: \"d02c7f84-f3da-432a-9cc3-2588b5256190\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.357457 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59k9x\" (UniqueName: \"kubernetes.io/projected/391ce6ca-1cf6-458a-a904-59997c7eb6db-kube-api-access-59k9x\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.357457 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.357349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmc4\" (UniqueName: \"kubernetes.io/projected/78788e11-73c4-423d-ac3a-d2b71d6f70b8-kube-api-access-lgmc4\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.358322 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.358309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-48rtn"
Sep 16 18:46:05.362099 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.362074    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-k5xrt\""
Sep 16 18:46:05.365060 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.365039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-48rtn"]
Sep 16 18:46:05.373555 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.373532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcrwp\" (UniqueName: \"kubernetes.io/projected/9761c3ca-de69-4601-98a8-bd173480628c-kube-api-access-qcrwp\") pod \"volume-data-source-validator-86695b7cc7-xgjxf\" (UID: \"9761c3ca-de69-4601-98a8-bd173480628c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf"
Sep 16 18:46:05.457720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-stats-auth\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.457720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d02c7f84-f3da-432a-9cc3-2588b5256190-serving-cert\") pod \"service-ca-operator-cf7c776b7-zgkp7\" (UID: \"d02c7f84-f3da-432a-9cc3-2588b5256190\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.457720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391ce6ca-1cf6-458a-a904-59997c7eb6db-trusted-ca\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.457720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/391ce6ca-1cf6-458a-a904-59997c7eb6db-config\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.458027 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/391ce6ca-1cf6-458a-a904-59997c7eb6db-serving-cert\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.458027 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.458027 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:05.457872    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:05.957847752 +0000 UTC m=+73.404942283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:05.458027 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxqkv\" (UniqueName: \"kubernetes.io/projected/d02c7f84-f3da-432a-9cc3-2588b5256190-kube-api-access-dxqkv\") pod \"service-ca-operator-cf7c776b7-zgkp7\" (UID: \"d02c7f84-f3da-432a-9cc3-2588b5256190\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.458027 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59k9x\" (UniqueName: \"kubernetes.io/projected/391ce6ca-1cf6-458a-a904-59997c7eb6db-kube-api-access-59k9x\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.458027 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmc4\" (UniqueName: \"kubernetes.io/projected/78788e11-73c4-423d-ac3a-d2b71d6f70b8-kube-api-access-lgmc4\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.458027 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.457974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9cb\" (UniqueName: \"kubernetes.io/projected/caf165e9-469b-427b-8af9-917a24b70ca8-kube-api-access-ks9cb\") pod \"network-check-source-6f8b768455-48rtn\" (UID: \"caf165e9-469b-427b-8af9-917a24b70ca8\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-48rtn"
Sep 16 18:46:05.458027 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.458004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-default-certificate\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.458411 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.458036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.458411 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.458057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d02c7f84-f3da-432a-9cc3-2588b5256190-config\") pod \"service-ca-operator-cf7c776b7-zgkp7\" (UID: \"d02c7f84-f3da-432a-9cc3-2588b5256190\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.458672 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:05.458424    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:05.458672 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:05.458513    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:05.958496005 +0000 UTC m=+73.405590533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : secret "router-metrics-certs-default" not found
Sep 16 18:46:05.458672 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.458624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/391ce6ca-1cf6-458a-a904-59997c7eb6db-config\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.458850 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.458669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d02c7f84-f3da-432a-9cc3-2588b5256190-config\") pod \"service-ca-operator-cf7c776b7-zgkp7\" (UID: \"d02c7f84-f3da-432a-9cc3-2588b5256190\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.458931 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.458906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391ce6ca-1cf6-458a-a904-59997c7eb6db-trusted-ca\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.460203 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.460179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d02c7f84-f3da-432a-9cc3-2588b5256190-serving-cert\") pod \"service-ca-operator-cf7c776b7-zgkp7\" (UID: \"d02c7f84-f3da-432a-9cc3-2588b5256190\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.460638 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.460620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-stats-auth\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.460827 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.460809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-default-certificate\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.460887 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.460866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/391ce6ca-1cf6-458a-a904-59997c7eb6db-serving-cert\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.461686 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.461673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf"
Sep 16 18:46:05.469549 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.469514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59k9x\" (UniqueName: \"kubernetes.io/projected/391ce6ca-1cf6-458a-a904-59997c7eb6db-kube-api-access-59k9x\") pod \"console-operator-696884fb9c-f6h98\" (UID: \"391ce6ca-1cf6-458a-a904-59997c7eb6db\") " pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.469674 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.469659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxqkv\" (UniqueName: \"kubernetes.io/projected/d02c7f84-f3da-432a-9cc3-2588b5256190-kube-api-access-dxqkv\") pod \"service-ca-operator-cf7c776b7-zgkp7\" (UID: \"d02c7f84-f3da-432a-9cc3-2588b5256190\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.472172 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.472115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmc4\" (UniqueName: \"kubernetes.io/projected/78788e11-73c4-423d-ac3a-d2b71d6f70b8-kube-api-access-lgmc4\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.559333 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.559291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9cb\" (UniqueName: \"kubernetes.io/projected/caf165e9-469b-427b-8af9-917a24b70ca8-kube-api-access-ks9cb\") pod \"network-check-source-6f8b768455-48rtn\" (UID: \"caf165e9-469b-427b-8af9-917a24b70ca8\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-48rtn"
Sep 16 18:46:05.568463 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.568438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9cb\" (UniqueName: \"kubernetes.io/projected/caf165e9-469b-427b-8af9-917a24b70ca8-kube-api-access-ks9cb\") pod \"network-check-source-6f8b768455-48rtn\" (UID: \"caf165e9-469b-427b-8af9-917a24b70ca8\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-48rtn"
Sep 16 18:46:05.571197 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.571181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"
Sep 16 18:46:05.579764 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.579744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:05.670907 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.670878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-48rtn"
Sep 16 18:46:05.746953 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.746927    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf"]
Sep 16 18:46:05.749060 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:05.749027    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9761c3ca_de69_4601_98a8_bd173480628c.slice/crio-fe4993e17fe8cf2a3a04ba0b27407904952118e525f45ba8c9ffa21e991f6f94 WatchSource:0}: Error finding container fe4993e17fe8cf2a3a04ba0b27407904952118e525f45ba8c9ffa21e991f6f94: Status 404 returned error can't find the container with id fe4993e17fe8cf2a3a04ba0b27407904952118e525f45ba8c9ffa21e991f6f94
Sep 16 18:46:05.842020 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.841923    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7"]
Sep 16 18:46:05.844347 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:05.844326    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02c7f84_f3da_432a_9cc3_2588b5256190.slice/crio-4ad226c02bf7a88d7c05240c418a74a14a78decb195b1d67d8b5efd8ec4d6932 WatchSource:0}: Error finding container 4ad226c02bf7a88d7c05240c418a74a14a78decb195b1d67d8b5efd8ec4d6932: Status 404 returned error can't find the container with id 4ad226c02bf7a88d7c05240c418a74a14a78decb195b1d67d8b5efd8ec4d6932
Sep 16 18:46:05.848843 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.848807    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-f6h98"]
Sep 16 18:46:05.852401 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:05.852353    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391ce6ca_1cf6_458a_a904_59997c7eb6db.slice/crio-f1e8c21d26f6e3a23b936fcf08856791893c757390d760db31b9e043c6993f0e WatchSource:0}: Error finding container f1e8c21d26f6e3a23b936fcf08856791893c757390d760db31b9e043c6993f0e: Status 404 returned error can't find the container with id f1e8c21d26f6e3a23b936fcf08856791893c757390d760db31b9e043c6993f0e
Sep 16 18:46:05.937463 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.937429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-48rtn"]
Sep 16 18:46:05.940038 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:05.940013    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf165e9_469b_427b_8af9_917a24b70ca8.slice/crio-e7b4e948ee4e7b6c0a0803a595484c80d673e3b22c590cc5e21175987c900cbc WatchSource:0}: Error finding container e7b4e948ee4e7b6c0a0803a595484c80d673e3b22c590cc5e21175987c900cbc: Status 404 returned error can't find the container with id e7b4e948ee4e7b6c0a0803a595484c80d673e3b22c590cc5e21175987c900cbc
Sep 16 18:46:05.961830 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.961800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.961947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:05.961859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:05.962015 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:05.961956    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:05.962015 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:05.961956    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:06.961936392 +0000 UTC m=+74.409030900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:05.962015 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:05.962001    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:06.961990704 +0000 UTC m=+74.409085213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : secret "router-metrics-certs-default" not found
Sep 16 18:46:06.436375 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:06.436333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7" event={"ID":"d02c7f84-f3da-432a-9cc3-2588b5256190","Type":"ContainerStarted","Data":"4ad226c02bf7a88d7c05240c418a74a14a78decb195b1d67d8b5efd8ec4d6932"}
Sep 16 18:46:06.438224 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:06.438165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf" event={"ID":"9761c3ca-de69-4601-98a8-bd173480628c","Type":"ContainerStarted","Data":"fe4993e17fe8cf2a3a04ba0b27407904952118e525f45ba8c9ffa21e991f6f94"}
Sep 16 18:46:06.439686 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:06.439658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-f6h98" event={"ID":"391ce6ca-1cf6-458a-a904-59997c7eb6db","Type":"ContainerStarted","Data":"f1e8c21d26f6e3a23b936fcf08856791893c757390d760db31b9e043c6993f0e"}
Sep 16 18:46:06.441919 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:06.441896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-48rtn" event={"ID":"caf165e9-469b-427b-8af9-917a24b70ca8","Type":"ContainerStarted","Data":"4aca6ac1aa5f673d0518b35631206904d3c566383c253f315bf10de6d2a62090"}
Sep 16 18:46:06.442035 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:06.441938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-48rtn" event={"ID":"caf165e9-469b-427b-8af9-917a24b70ca8","Type":"ContainerStarted","Data":"e7b4e948ee4e7b6c0a0803a595484c80d673e3b22c590cc5e21175987c900cbc"}
Sep 16 18:46:06.460533 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:06.460464    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-48rtn" podStartSLOduration=1.460444413 podStartE2EDuration="1.460444413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:06.45894951 +0000 UTC m=+73.906044065" watchObservedRunningTime="2026-09-16 18:46:06.460444413 +0000 UTC m=+73.907538935"
Sep 16 18:46:06.971077 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:06.971047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:06.971208 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:06.971131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:06.971280 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:06.971209    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:06.971336 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:06.971285    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:08.971264377 +0000 UTC m=+76.418358901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : secret "router-metrics-certs-default" not found
Sep 16 18:46:06.971336 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:06.971307    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:08.97129627 +0000 UTC m=+76.418390778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:07.446553 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:07.446521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf" event={"ID":"9761c3ca-de69-4601-98a8-bd173480628c","Type":"ContainerStarted","Data":"56eecaf22995041035c6c37e06de51ef46fd28b202e04c7244c4665348dc69e0"}
Sep 16 18:46:07.463103 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:07.463057    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-xgjxf" podStartSLOduration=1.296802105 podStartE2EDuration="2.463035515s" totalImagesPullingTime="1.16623341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:07.461854016 +0000 UTC m=+74.908948569" watchObservedRunningTime="2026-09-16 18:46:07.463035515 +0000 UTC m=+74.910130047"
Sep 16 18:46:08.987109 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:08.987009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:08.987109 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:08.987089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:08.987574 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:08.987172    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:12.98715523 +0000 UTC m=+80.434249738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:08.987574 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:08.987228    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:08.987574 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:08.987283    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:12.987270195 +0000 UTC m=+80.434364703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : secret "router-metrics-certs-default" not found
Sep 16 18:46:09.458270 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:09.458239    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/0.log"
Sep 16 18:46:09.458454 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:09.458282    2578 generic.go:356] "Generic (PLEG): container finished" podID="391ce6ca-1cf6-458a-a904-59997c7eb6db" containerID="043d04ab79a8efae97778335528b27e58b8623f2b9e1ccf5cae4a02a5d79b557" exitCode=255
Sep 16 18:46:09.458454 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:09.458372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-f6h98" event={"ID":"391ce6ca-1cf6-458a-a904-59997c7eb6db","Type":"ContainerDied","Data":"043d04ab79a8efae97778335528b27e58b8623f2b9e1ccf5cae4a02a5d79b557"}
Sep 16 18:46:09.458607 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:09.458589    2578 scope.go:117] "RemoveContainer" containerID="043d04ab79a8efae97778335528b27e58b8623f2b9e1ccf5cae4a02a5d79b557"
Sep 16 18:46:09.459907 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:09.459880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7" event={"ID":"d02c7f84-f3da-432a-9cc3-2588b5256190","Type":"ContainerStarted","Data":"53807f0e6e1a5e73b2664d3f33d55660bc26181a42efe2cd52eb2dfe4c031d63"}
Sep 16 18:46:09.505119 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:09.504989    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7" podStartSLOduration=1.7568701340000001 podStartE2EDuration="4.50496836s" totalImagesPullingTime="2.748098226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:09.504502126 +0000 UTC m=+76.951596676" watchObservedRunningTime="2026-09-16 18:46:09.50496836 +0000 UTC m=+76.952062889"
Sep 16 18:46:10.466852 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:10.466826    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 18:46:10.467302 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:10.467286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/0.log"
Sep 16 18:46:10.467389 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:10.467327    2578 generic.go:356] "Generic (PLEG): container finished" podID="391ce6ca-1cf6-458a-a904-59997c7eb6db" containerID="de4ebcc64e18b331373e1b3da66f8a88d2ac6df9abdccd4b02f5586c1d3fd5b8" exitCode=255
Sep 16 18:46:10.467443 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:10.467398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-f6h98" event={"ID":"391ce6ca-1cf6-458a-a904-59997c7eb6db","Type":"ContainerDied","Data":"de4ebcc64e18b331373e1b3da66f8a88d2ac6df9abdccd4b02f5586c1d3fd5b8"}
Sep 16 18:46:10.467507 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:10.467452    2578 scope.go:117] "RemoveContainer" containerID="043d04ab79a8efae97778335528b27e58b8623f2b9e1ccf5cae4a02a5d79b557"
Sep 16 18:46:10.467709 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:10.467686    2578 scope.go:117] "RemoveContainer" containerID="de4ebcc64e18b331373e1b3da66f8a88d2ac6df9abdccd4b02f5586c1d3fd5b8"
Sep 16 18:46:10.467888 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:10.467869    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-f6h98_openshift-console-operator(391ce6ca-1cf6-458a-a904-59997c7eb6db)\"" pod="openshift-console-operator/console-operator-696884fb9c-f6h98" podUID="391ce6ca-1cf6-458a-a904-59997c7eb6db"
Sep 16 18:46:11.471462 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:11.471436    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 18:46:11.471957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:11.471865    2578 scope.go:117] "RemoveContainer" containerID="de4ebcc64e18b331373e1b3da66f8a88d2ac6df9abdccd4b02f5586c1d3fd5b8"
Sep 16 18:46:11.472071 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:11.472050    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-f6h98_openshift-console-operator(391ce6ca-1cf6-458a-a904-59997c7eb6db)\"" pod="openshift-console-operator/console-operator-696884fb9c-f6h98" podUID="391ce6ca-1cf6-458a-a904-59997c7eb6db"
Sep 16 18:46:12.022801 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.022766    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hgxv5"]
Sep 16 18:46:12.026970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.026949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.029944 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.029924    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:46:12.031169 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.031150    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jtph7\""
Sep 16 18:46:12.031293 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.031237    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:46:12.031385 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.031357    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:46:12.031385 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.031369    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:46:12.036266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.036245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hgxv5"]
Sep 16 18:46:12.110840 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.110806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32a20d55-2e99-4f54-9474-9890a6af967c-signing-cabundle\") pod \"service-ca-568869fdc5-hgxv5\" (UID: \"32a20d55-2e99-4f54-9474-9890a6af967c\") " pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.110840 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.110845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgb5\" (UniqueName: \"kubernetes.io/projected/32a20d55-2e99-4f54-9474-9890a6af967c-kube-api-access-mfgb5\") pod \"service-ca-568869fdc5-hgxv5\" (UID: \"32a20d55-2e99-4f54-9474-9890a6af967c\") " pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.111073 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.110876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32a20d55-2e99-4f54-9474-9890a6af967c-signing-key\") pod \"service-ca-568869fdc5-hgxv5\" (UID: \"32a20d55-2e99-4f54-9474-9890a6af967c\") " pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.212103 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.212060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32a20d55-2e99-4f54-9474-9890a6af967c-signing-cabundle\") pod \"service-ca-568869fdc5-hgxv5\" (UID: \"32a20d55-2e99-4f54-9474-9890a6af967c\") " pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.212103 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.212101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgb5\" (UniqueName: \"kubernetes.io/projected/32a20d55-2e99-4f54-9474-9890a6af967c-kube-api-access-mfgb5\") pod \"service-ca-568869fdc5-hgxv5\" (UID: \"32a20d55-2e99-4f54-9474-9890a6af967c\") " pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.212318 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.212121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32a20d55-2e99-4f54-9474-9890a6af967c-signing-key\") pod \"service-ca-568869fdc5-hgxv5\" (UID: \"32a20d55-2e99-4f54-9474-9890a6af967c\") " pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.212784 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.212760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/32a20d55-2e99-4f54-9474-9890a6af967c-signing-cabundle\") pod \"service-ca-568869fdc5-hgxv5\" (UID: \"32a20d55-2e99-4f54-9474-9890a6af967c\") " pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.214592 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.214570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/32a20d55-2e99-4f54-9474-9890a6af967c-signing-key\") pod \"service-ca-568869fdc5-hgxv5\" (UID: \"32a20d55-2e99-4f54-9474-9890a6af967c\") " pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.222555 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.222533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgb5\" (UniqueName: \"kubernetes.io/projected/32a20d55-2e99-4f54-9474-9890a6af967c-kube-api-access-mfgb5\") pod \"service-ca-568869fdc5-hgxv5\" (UID: \"32a20d55-2e99-4f54-9474-9890a6af967c\") " pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.268372 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.268339    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"]
Sep 16 18:46:12.271292 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.271275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:12.273906 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.273856    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:46:12.274182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.274162    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gp6n9\""
Sep 16 18:46:12.275373 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.275354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:46:12.285574 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.285553    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"]
Sep 16 18:46:12.312523 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.312470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:12.312523 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.312525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:12.335468 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.335447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-hgxv5"
Sep 16 18:46:12.362390 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.362359    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-67rrr"]
Sep 16 18:46:12.367098 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.367072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.371024 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.370998    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:46:12.371159 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.371073    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:46:12.371159 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.371139    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hz5t4\""
Sep 16 18:46:12.371322 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.371080    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:46:12.371447 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.371430    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:46:12.375821 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.375798    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-67rrr"]
Sep 16 18:46:12.413447 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.413417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6afc2843-18bc-4f0f-a625-3a803fbf76cc-data-volume\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.413632 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.413455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwwxc\" (UniqueName: \"kubernetes.io/projected/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-api-access-wwwxc\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.413632 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.413523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6afc2843-18bc-4f0f-a625-3a803fbf76cc-crio-socket\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.413632 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.413575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:12.413632 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.413607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:12.413822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.413634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.413822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.413654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.413822 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:12.413716    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:12.413822 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:12.413782    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert podName:cc160e38-cf9b-49a1-be39-0afb8eaca6ac nodeName:}" failed. No retries permitted until 2026-09-16 18:46:12.913766553 +0000 UTC m=+80.360861066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6k7dk" (UID: "cc160e38-cf9b-49a1-be39-0afb8eaca6ac") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:12.414287 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.414266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:12.514874 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.514840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.514909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6afc2843-18bc-4f0f-a625-3a803fbf76cc-data-volume\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.514936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwwxc\" (UniqueName: \"kubernetes.io/projected/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-api-access-wwwxc\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:12.514965    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.514972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6afc2843-18bc-4f0f-a625-3a803fbf76cc-crio-socket\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.515036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6afc2843-18bc-4f0f-a625-3a803fbf76cc-crio-socket\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:12.515041    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.015020424 +0000 UTC m=+80.462114948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.515154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:12.515255    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.515281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6afc2843-18bc-4f0f-a625-3a803fbf76cc-data-volume\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.515335 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:12.515311    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.01529527 +0000 UTC m=+80.462389783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:12.539910 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.539892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwwxc\" (UniqueName: \"kubernetes.io/projected/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-api-access-wwwxc\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:12.610225 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.610059    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hgxv5"]
Sep 16 18:46:12.612682 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:12.612655    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a20d55_2e99_4f54_9474_9890a6af967c.slice/crio-a6065acf0de76d6a6c8dc5375413b76cfbc20b136810293299f8a87afa19b2b5 WatchSource:0}: Error finding container a6065acf0de76d6a6c8dc5375413b76cfbc20b136810293299f8a87afa19b2b5: Status 404 returned error can't find the container with id a6065acf0de76d6a6c8dc5375413b76cfbc20b136810293299f8a87afa19b2b5
Sep 16 18:46:12.917989 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:12.917910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:12.918208 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:12.918021    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:12.918208 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:12.918069    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert podName:cc160e38-cf9b-49a1-be39-0afb8eaca6ac nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.918056975 +0000 UTC m=+81.365151482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6k7dk" (UID: "cc160e38-cf9b-49a1-be39-0afb8eaca6ac") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:13.019198 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:13.019164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:13.019198 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:13.019204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:13.019458 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:13.019225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:13.019458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:13.019288    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:13.019458 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:13.019291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:13.019458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:13.019340    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.019324047 +0000 UTC m=+81.466418578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:13.019458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:13.019384    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:21.019378631 +0000 UTC m=+88.466473139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:13.019458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:13.019394    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:13.019458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:13.019448    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs podName:78788e11-73c4-423d-ac3a-d2b71d6f70b8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:21.019430654 +0000 UTC m=+88.466525168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs") pod "router-default-7df8cfc95d-tzm6f" (UID: "78788e11-73c4-423d-ac3a-d2b71d6f70b8") : secret "router-metrics-certs-default" not found
Sep 16 18:46:13.019458 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:13.019454    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:13.019779 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:13.019524    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.019510339 +0000 UTC m=+81.466604851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:13.479190 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:13.479153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-hgxv5" event={"ID":"32a20d55-2e99-4f54-9474-9890a6af967c","Type":"ContainerStarted","Data":"abe434d2af1daead82fbd6e64050a8e918591c4155b56ad8489365d56fa75dbb"}
Sep 16 18:46:13.479363 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:13.479206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-hgxv5" event={"ID":"32a20d55-2e99-4f54-9474-9890a6af967c","Type":"ContainerStarted","Data":"a6065acf0de76d6a6c8dc5375413b76cfbc20b136810293299f8a87afa19b2b5"}
Sep 16 18:46:13.497146 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:13.497102    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-hgxv5" podStartSLOduration=1.497086599 podStartE2EDuration="1.497086599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:13.496071868 +0000 UTC m=+80.943166399" watchObservedRunningTime="2026-09-16 18:46:13.497086599 +0000 UTC m=+80.944181126"
Sep 16 18:46:13.926625 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:13.926584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:13.927144 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:13.926749    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:13.927144 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:13.926839    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert podName:cc160e38-cf9b-49a1-be39-0afb8eaca6ac nodeName:}" failed. No retries permitted until 2026-09-16 18:46:15.926808317 +0000 UTC m=+83.373902828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6k7dk" (UID: "cc160e38-cf9b-49a1-be39-0afb8eaca6ac") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:14.027640 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:14.027596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:14.027839 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:14.027648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:14.027839 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:14.027790    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:14.027953 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:14.027849    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.02783081 +0000 UTC m=+83.474925324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:14.028261 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:14.028238    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:14.028329 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:14.028310    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.028293533 +0000 UTC m=+83.475388115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:14.928339 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:14.928300    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9pgsd_b7f9ed7d-0ea5-4c26-b542-703da8f84edb/dns-node-resolver/0.log"
Sep 16 18:46:15.579826 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:15.579795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:15.580090 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:15.580071    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:15.580191 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:15.580133    2578 scope.go:117] "RemoveContainer" containerID="de4ebcc64e18b331373e1b3da66f8a88d2ac6df9abdccd4b02f5586c1d3fd5b8"
Sep 16 18:46:15.580294 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:15.580277    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-f6h98_openshift-console-operator(391ce6ca-1cf6-458a-a904-59997c7eb6db)\"" pod="openshift-console-operator/console-operator-696884fb9c-f6h98" podUID="391ce6ca-1cf6-458a-a904-59997c7eb6db"
Sep 16 18:46:15.944177 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:15.944097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:15.944551 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:15.944242    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:15.944551 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:15.944305    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert podName:cc160e38-cf9b-49a1-be39-0afb8eaca6ac nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.944291067 +0000 UTC m=+87.391385576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6k7dk" (UID: "cc160e38-cf9b-49a1-be39-0afb8eaca6ac") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:16.045628 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:16.045593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:16.045628 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:16.045630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:16.045833 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:16.045715    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:16.045833 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:16.045751    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:16.045833 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:16.045767    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.045753135 +0000 UTC m=+87.492847644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:16.045833 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:16.045820    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.04580195 +0000 UTC m=+87.492896462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:16.328241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:16.328213    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zh85g_acb7ae1c-13d5-4745-aa4d-8e5f951e8f15/node-ca/0.log"
Sep 16 18:46:16.488078 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:16.488048    2578 scope.go:117] "RemoveContainer" containerID="de4ebcc64e18b331373e1b3da66f8a88d2ac6df9abdccd4b02f5586c1d3fd5b8"
Sep 16 18:46:16.488261 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:16.488194    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-f6h98_openshift-console-operator(391ce6ca-1cf6-458a-a904-59997c7eb6db)\"" pod="openshift-console-operator/console-operator-696884fb9c-f6h98" podUID="391ce6ca-1cf6-458a-a904-59997c7eb6db"
Sep 16 18:46:16.728960 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:16.728888    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-djtj7_5fe145cc-75f4-4d0e-844e-4822424e1071/serve-healthcheck-canary/0.log"
Sep 16 18:46:19.981211 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:19.981174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:19.981610 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:19.981326    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:19.981610 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:19.981392    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert podName:cc160e38-cf9b-49a1-be39-0afb8eaca6ac nodeName:}" failed. No retries permitted until 2026-09-16 18:46:27.981376708 +0000 UTC m=+95.428471216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6k7dk" (UID: "cc160e38-cf9b-49a1-be39-0afb8eaca6ac") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:20.081829 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:20.081792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:20.081829 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:20.081830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:20.081993 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:20.081923    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:20.081993 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:20.081946    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:20.081993 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:20.081990    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:28.081971578 +0000 UTC m=+95.529066100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:20.082085 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:20.082005    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls podName:6afc2843-18bc-4f0f-a625-3a803fbf76cc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:28.081998846 +0000 UTC m=+95.529093354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67rrr" (UID: "6afc2843-18bc-4f0f-a625-3a803fbf76cc") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:21.088998 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:21.088958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:21.089505 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:21.089025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:21.089684 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:21.089666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78788e11-73c4-423d-ac3a-d2b71d6f70b8-service-ca-bundle\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:21.091508 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:21.091471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78788e11-73c4-423d-ac3a-d2b71d6f70b8-metrics-certs\") pod \"router-default-7df8cfc95d-tzm6f\" (UID: \"78788e11-73c4-423d-ac3a-d2b71d6f70b8\") " pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:21.184221 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:21.184171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:21.505931 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:21.505817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7df8cfc95d-tzm6f"]
Sep 16 18:46:21.508225 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:21.508197    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78788e11_73c4_423d_ac3a_d2b71d6f70b8.slice/crio-3f26c0fb4da814758491121db5a2e28d5e4aa9e5c125086d749b224ebb297bc0 WatchSource:0}: Error finding container 3f26c0fb4da814758491121db5a2e28d5e4aa9e5c125086d749b224ebb297bc0: Status 404 returned error can't find the container with id 3f26c0fb4da814758491121db5a2e28d5e4aa9e5c125086d749b224ebb297bc0
Sep 16 18:46:22.509154 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:22.509108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7df8cfc95d-tzm6f" event={"ID":"78788e11-73c4-423d-ac3a-d2b71d6f70b8","Type":"ContainerStarted","Data":"ab582c1357428d4e13103ecaf5a645ae49fbc8b281c6e564635da7eee84221f7"}
Sep 16 18:46:22.509563 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:22.509166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7df8cfc95d-tzm6f" event={"ID":"78788e11-73c4-423d-ac3a-d2b71d6f70b8","Type":"ContainerStarted","Data":"3f26c0fb4da814758491121db5a2e28d5e4aa9e5c125086d749b224ebb297bc0"}
Sep 16 18:46:22.538312 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:22.538258    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7df8cfc95d-tzm6f" podStartSLOduration=17.538238995 podStartE2EDuration="17.538238995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:22.537655055 +0000 UTC m=+89.984749606" watchObservedRunningTime="2026-09-16 18:46:22.538238995 +0000 UTC m=+89.985333525"
Sep 16 18:46:23.185183 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:23.185152    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:23.187758 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:23.187735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:23.512044 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:23.511963    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:23.513315 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:23.513294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7df8cfc95d-tzm6f"
Sep 16 18:46:28.045520 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.045441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:28.047820 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.047795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cc160e38-cf9b-49a1-be39-0afb8eaca6ac-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6k7dk\" (UID: \"cc160e38-cf9b-49a1-be39-0afb8eaca6ac\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:28.146061 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.146024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:28.146061 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.146061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:28.146687 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.146658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6afc2843-18bc-4f0f-a625-3a803fbf76cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:28.148344 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.148324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6afc2843-18bc-4f0f-a625-3a803fbf76cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67rrr\" (UID: \"6afc2843-18bc-4f0f-a625-3a803fbf76cc\") " pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:28.180336 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.180296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"
Sep 16 18:46:28.277875 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.277846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-67rrr"
Sep 16 18:46:28.450532 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.450506    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk"]
Sep 16 18:46:28.452846 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:28.452824    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc160e38_cf9b_49a1_be39_0afb8eaca6ac.slice/crio-a03dc484a4c003c31d01e9866b464f0cbc6d1f2a81f1a2f5ffe97da01c45819a WatchSource:0}: Error finding container a03dc484a4c003c31d01e9866b464f0cbc6d1f2a81f1a2f5ffe97da01c45819a: Status 404 returned error can't find the container with id a03dc484a4c003c31d01e9866b464f0cbc6d1f2a81f1a2f5ffe97da01c45819a
Sep 16 18:46:28.529468 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.529425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk" event={"ID":"cc160e38-cf9b-49a1-be39-0afb8eaca6ac","Type":"ContainerStarted","Data":"a03dc484a4c003c31d01e9866b464f0cbc6d1f2a81f1a2f5ffe97da01c45819a"}
Sep 16 18:46:28.559202 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:28.559169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-67rrr"]
Sep 16 18:46:28.561691 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:28.561669    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afc2843_18bc_4f0f_a625_3a803fbf76cc.slice/crio-ac627efefabd88a071da202fa177b3e2ebe71418fbf0053386bffa388a82f835 WatchSource:0}: Error finding container ac627efefabd88a071da202fa177b3e2ebe71418fbf0053386bffa388a82f835: Status 404 returned error can't find the container with id ac627efefabd88a071da202fa177b3e2ebe71418fbf0053386bffa388a82f835
Sep 16 18:46:29.058688 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:29.058656    2578 scope.go:117] "RemoveContainer" containerID="de4ebcc64e18b331373e1b3da66f8a88d2ac6df9abdccd4b02f5586c1d3fd5b8"
Sep 16 18:46:29.535488 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:29.535424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67rrr" event={"ID":"6afc2843-18bc-4f0f-a625-3a803fbf76cc","Type":"ContainerStarted","Data":"47e4c2c3e0735f4bc32a7c2dffb63364c33620e8f84886f434fee92f512fa819"}
Sep 16 18:46:29.535681 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:29.535516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67rrr" event={"ID":"6afc2843-18bc-4f0f-a625-3a803fbf76cc","Type":"ContainerStarted","Data":"c992ba170284e784252c07e5211a676b0e9c42f337bc12e8c3162f4988fdf2f5"}
Sep 16 18:46:29.535681 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:29.535549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67rrr" event={"ID":"6afc2843-18bc-4f0f-a625-3a803fbf76cc","Type":"ContainerStarted","Data":"ac627efefabd88a071da202fa177b3e2ebe71418fbf0053386bffa388a82f835"}
Sep 16 18:46:29.541371 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:29.541342    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 18:46:29.541538 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:29.541407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-f6h98" event={"ID":"391ce6ca-1cf6-458a-a904-59997c7eb6db","Type":"ContainerStarted","Data":"054c8bd34905a253b559f955d187d90037def6da46ce370e2f8efbf51c04c657"}
Sep 16 18:46:29.541831 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:29.541786    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:29.560019 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:29.559964    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-f6h98" podStartSLOduration=21.817803669 podStartE2EDuration="24.55994532s" totalImagesPullingTime="2.742141651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:29.558810997 +0000 UTC m=+97.005905559" watchObservedRunningTime="2026-09-16 18:46:29.55994532 +0000 UTC m=+97.007039851"
Sep 16 18:46:29.580063 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:29.579979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-f6h98"
Sep 16 18:46:30.546307 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:30.546273    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk" event={"ID":"cc160e38-cf9b-49a1-be39-0afb8eaca6ac","Type":"ContainerStarted","Data":"3c21ba546a64d32993fae66c758487d128c6b72e674ea689655717ba5f3e8615"}
Sep 16 18:46:30.562441 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:30.562400    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6k7dk" podStartSLOduration=17.369960785 podStartE2EDuration="18.562381255s" totalImagesPullingTime="1.19242047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:30.561965746 +0000 UTC m=+98.009060275" watchObservedRunningTime="2026-09-16 18:46:30.562381255 +0000 UTC m=+98.009475819"
Sep 16 18:46:31.552247 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:31.552215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67rrr" event={"ID":"6afc2843-18bc-4f0f-a625-3a803fbf76cc","Type":"ContainerStarted","Data":"6479c70ddcc22187d48774f6e989ca7442b58e719bbb8d3da1fa2e174be2056c"}
Sep 16 18:46:31.580640 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:31.580593    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-67rrr" podStartSLOduration=17.726518989 podStartE2EDuration="19.580574019s" totalImagesPullingTime="1.85405503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:31.578963798 +0000 UTC m=+99.026058348" watchObservedRunningTime="2026-09-16 18:46:31.580574019 +0000 UTC m=+99.027668549"
Sep 16 18:46:34.431328 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.431296    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n5x8n"
Sep 16 18:46:34.652191 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.652121    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"]
Sep 16 18:46:34.683902 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.683871    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-gl9hm"]
Sep 16 18:46:34.684066 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.684002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"
Sep 16 18:46:34.687196 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.687166    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tzqr5\""
Sep 16 18:46:34.687582 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.687560    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:46:34.697683 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.697660    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"]
Sep 16 18:46:34.697683 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.697693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-gl9hm"]
Sep 16 18:46:34.697863 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.697775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-gl9hm"
Sep 16 18:46:34.702957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.702939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rlnb9\""
Sep 16 18:46:34.703177 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.703162    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:46:34.703242 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.703204    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:46:34.802032 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.801998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cdcbfc31-d826-47fe-870e-3ca14c6ef6bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-zqmjb\" (UID: \"cdcbfc31-d826-47fe-870e-3ca14c6ef6bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"
Sep 16 18:46:34.802207 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.802083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5vz\" (UniqueName: \"kubernetes.io/projected/2ca91436-3300-447c-a199-5e12d7b3f511-kube-api-access-zp5vz\") pod \"downloads-76f99584f4-gl9hm\" (UID: \"2ca91436-3300-447c-a199-5e12d7b3f511\") " pod="openshift-console/downloads-76f99584f4-gl9hm"
Sep 16 18:46:34.903400 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.903318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cdcbfc31-d826-47fe-870e-3ca14c6ef6bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-zqmjb\" (UID: \"cdcbfc31-d826-47fe-870e-3ca14c6ef6bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"
Sep 16 18:46:34.903585 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.903411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5vz\" (UniqueName: \"kubernetes.io/projected/2ca91436-3300-447c-a199-5e12d7b3f511-kube-api-access-zp5vz\") pod \"downloads-76f99584f4-gl9hm\" (UID: \"2ca91436-3300-447c-a199-5e12d7b3f511\") " pod="openshift-console/downloads-76f99584f4-gl9hm"
Sep 16 18:46:34.905773 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.905749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cdcbfc31-d826-47fe-870e-3ca14c6ef6bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-zqmjb\" (UID: \"cdcbfc31-d826-47fe-870e-3ca14c6ef6bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"
Sep 16 18:46:34.914533 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.914510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5vz\" (UniqueName: \"kubernetes.io/projected/2ca91436-3300-447c-a199-5e12d7b3f511-kube-api-access-zp5vz\") pod \"downloads-76f99584f4-gl9hm\" (UID: \"2ca91436-3300-447c-a199-5e12d7b3f511\") " pod="openshift-console/downloads-76f99584f4-gl9hm"
Sep 16 18:46:34.994065 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:34.994027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"
Sep 16 18:46:35.006326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.006303    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-gl9hm"
Sep 16 18:46:35.205848 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.205753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:46:35.208047 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.208026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e084045-524d-49ab-9191-7a4c3430cf0c-metrics-tls\") pod \"dns-default-h7lbj\" (UID: \"0e084045-524d-49ab-9191-7a4c3430cf0c\") " pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:46:35.255066 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.255038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dwfv6\""
Sep 16 18:46:35.263145 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.263112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:46:35.280060 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.279872    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"]
Sep 16 18:46:35.282057 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:35.282028    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcbfc31_d826_47fe_870e_3ca14c6ef6bc.slice/crio-0dfbf93aed812a492c6a042a914bd9711a047bfb5637a2255a2fe420e2ebb7f0 WatchSource:0}: Error finding container 0dfbf93aed812a492c6a042a914bd9711a047bfb5637a2255a2fe420e2ebb7f0: Status 404 returned error can't find the container with id 0dfbf93aed812a492c6a042a914bd9711a047bfb5637a2255a2fe420e2ebb7f0
Sep 16 18:46:35.299638 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.299550    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-gl9hm"]
Sep 16 18:46:35.302458 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:35.302429    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca91436_3300_447c_a199_5e12d7b3f511.slice/crio-74277ca2a48c016f24dd194eb98251253dd14c62e3b642ae2c4ae60e072a76b9 WatchSource:0}: Error finding container 74277ca2a48c016f24dd194eb98251253dd14c62e3b642ae2c4ae60e072a76b9: Status 404 returned error can't find the container with id 74277ca2a48c016f24dd194eb98251253dd14c62e3b642ae2c4ae60e072a76b9
Sep 16 18:46:35.540127 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.540102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h7lbj"]
Sep 16 18:46:35.542288 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:35.542258    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e084045_524d_49ab_9191_7a4c3430cf0c.slice/crio-9278db76a6608fd4c6c2c0081b259fc5beae699a9e34073712489a6a9ad81ac9 WatchSource:0}: Error finding container 9278db76a6608fd4c6c2c0081b259fc5beae699a9e34073712489a6a9ad81ac9: Status 404 returned error can't find the container with id 9278db76a6608fd4c6c2c0081b259fc5beae699a9e34073712489a6a9ad81ac9
Sep 16 18:46:35.566242 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.566207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h7lbj" event={"ID":"0e084045-524d-49ab-9191-7a4c3430cf0c","Type":"ContainerStarted","Data":"9278db76a6608fd4c6c2c0081b259fc5beae699a9e34073712489a6a9ad81ac9"}
Sep 16 18:46:35.567296 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.567269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-gl9hm" event={"ID":"2ca91436-3300-447c-a199-5e12d7b3f511","Type":"ContainerStarted","Data":"74277ca2a48c016f24dd194eb98251253dd14c62e3b642ae2c4ae60e072a76b9"}
Sep 16 18:46:35.568371 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:35.568346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb" event={"ID":"cdcbfc31-d826-47fe-870e-3ca14c6ef6bc","Type":"ContainerStarted","Data":"0dfbf93aed812a492c6a042a914bd9711a047bfb5637a2255a2fe420e2ebb7f0"}
Sep 16 18:46:37.579571 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:37.579429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h7lbj" event={"ID":"0e084045-524d-49ab-9191-7a4c3430cf0c","Type":"ContainerStarted","Data":"5c30cc250b745cf56c2b02feaab8da6be2584d27216345d3d2b2de1103d47bfc"}
Sep 16 18:46:37.579571 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:37.579507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h7lbj" event={"ID":"0e084045-524d-49ab-9191-7a4c3430cf0c","Type":"ContainerStarted","Data":"c2cf308deba8ade5ac9433a7ced13bacc2e10f8adadb7c49f13f4b42c9d99dc2"}
Sep 16 18:46:37.579571 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:37.579538    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:46:37.581316 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:37.581276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb" event={"ID":"cdcbfc31-d826-47fe-870e-3ca14c6ef6bc","Type":"ContainerStarted","Data":"1440cc8a712a762aaef220fafcef5e0a269982c23578afb205bdb1362ae7de37"}
Sep 16 18:46:37.581525 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:37.581497    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"
Sep 16 18:46:37.586614 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:37.586588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb"
Sep 16 18:46:37.612875 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:37.612810    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h7lbj" podStartSLOduration=64.897510221 podStartE2EDuration="1m6.612788368s" totalImagesPullingTime="1.715278147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:37.611277004 +0000 UTC m=+105.058371567" watchObservedRunningTime="2026-09-16 18:46:37.612788368 +0000 UTC m=+105.059882900"
Sep 16 18:46:37.629375 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:37.629323    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zqmjb" podStartSLOduration=2.3826105699999998 podStartE2EDuration="3.629303753s" totalImagesPullingTime="1.246693183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:37.628243026 +0000 UTC m=+105.075337584" watchObservedRunningTime="2026-09-16 18:46:37.629303753 +0000 UTC m=+105.076398283"
Sep 16 18:46:38.386225 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.386187    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"]
Sep 16 18:46:38.388619 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.388591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.391701 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.391679    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:46:38.392915 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.392892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:46:38.393026 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.392987    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:46:38.393126 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.393030    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gtjzd\""
Sep 16 18:46:38.393429 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.393270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:46:38.393429 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.393326    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:46:38.400779 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.400753    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"]
Sep 16 18:46:38.540942 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.540899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdgp\" (UniqueName: \"kubernetes.io/projected/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-kube-api-access-hfdgp\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.541132 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.540973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.541132 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.541028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.541132 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.541072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.641916 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.641826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdgp\" (UniqueName: \"kubernetes.io/projected/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-kube-api-access-hfdgp\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.641916 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.641913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.642405 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.641967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.642405 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.642010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.642405 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:38.642119    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 18:46:38.642405 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:38.642194    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-prometheus-operator-tls podName:49729b4e-8f6c-4b46-8cb0-8cae82929ab5 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:39.142176827 +0000 UTC m=+106.589271341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-w9kfm" (UID: "49729b4e-8f6c-4b46-8cb0-8cae82929ab5") : secret "prometheus-operator-tls" not found
Sep 16 18:46:38.642757 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.642713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.644958 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.644937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:38.653618 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:38.653592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdgp\" (UniqueName: \"kubernetes.io/projected/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-kube-api-access-hfdgp\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:39.146315 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.146277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:39.148897 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.148864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/49729b4e-8f6c-4b46-8cb0-8cae82929ab5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-w9kfm\" (UID: \"49729b4e-8f6c-4b46-8cb0-8cae82929ab5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:39.302514 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.302466    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"
Sep 16 18:46:39.584512 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.584341    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm"]
Sep 16 18:46:39.587350 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:39.587324    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49729b4e_8f6c_4b46_8cb0_8cae82929ab5.slice/crio-c3a9d3994bb749c7741740c2bb4799dacf8716971a3b517c1fee3bb5cef83bec WatchSource:0}: Error finding container c3a9d3994bb749c7741740c2bb4799dacf8716971a3b517c1fee3bb5cef83bec: Status 404 returned error can't find the container with id c3a9d3994bb749c7741740c2bb4799dacf8716971a3b517c1fee3bb5cef83bec
Sep 16 18:46:39.883444 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.883362    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-549d46475c-z29xr"]
Sep 16 18:46:39.886608 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.886589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:39.889585 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.889565    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:46:39.889792 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.889774    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:46:39.891036 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.891016    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l48gs\""
Sep 16 18:46:39.891141 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.891038    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:46:39.891141 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.891015    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:46:39.891141 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.891025    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:46:39.899101 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:39.899079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549d46475c-z29xr"]
Sep 16 18:46:40.055237 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.055197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-config\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.058506 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.055249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-serving-cert\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.058506 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.055676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qc2c\" (UniqueName: \"kubernetes.io/projected/f38c5fef-047e-45b3-bafe-feca63d7b4c9-kube-api-access-5qc2c\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.058506 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.055854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-oauth-config\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.058506 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.055896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-service-ca\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.058506 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.056114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-oauth-serving-cert\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.157342 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.157256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-oauth-serving-cert\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.157342 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.157309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-config\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.157619 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.157351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-serving-cert\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.157619 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.157566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qc2c\" (UniqueName: \"kubernetes.io/projected/f38c5fef-047e-45b3-bafe-feca63d7b4c9-kube-api-access-5qc2c\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.157725 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.157634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-oauth-config\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.157725 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.157672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-service-ca\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.158171 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.158143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-oauth-serving-cert\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.158329 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.158307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-service-ca\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.158793 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.158767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-config\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.160582 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.160558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-oauth-config\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.160686 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.160642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-serving-cert\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.166798 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.166756    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qc2c\" (UniqueName: \"kubernetes.io/projected/f38c5fef-047e-45b3-bafe-feca63d7b4c9-kube-api-access-5qc2c\") pod \"console-549d46475c-z29xr\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") " pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.196629 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.196581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:40.484013 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.483936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549d46475c-z29xr"]
Sep 16 18:46:40.598945 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:40.598904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm" event={"ID":"49729b4e-8f6c-4b46-8cb0-8cae82929ab5","Type":"ContainerStarted","Data":"c3a9d3994bb749c7741740c2bb4799dacf8716971a3b517c1fee3bb5cef83bec"}
Sep 16 18:46:40.871574 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:40.871528    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38c5fef_047e_45b3_bafe_feca63d7b4c9.slice/crio-168ec95417b8a1ec3dd1ac480db44295a80e989c118a74d6b714431afd305ecf WatchSource:0}: Error finding container 168ec95417b8a1ec3dd1ac480db44295a80e989c118a74d6b714431afd305ecf: Status 404 returned error can't find the container with id 168ec95417b8a1ec3dd1ac480db44295a80e989c118a74d6b714431afd305ecf
Sep 16 18:46:41.606267 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:41.606224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549d46475c-z29xr" event={"ID":"f38c5fef-047e-45b3-bafe-feca63d7b4c9","Type":"ContainerStarted","Data":"168ec95417b8a1ec3dd1ac480db44295a80e989c118a74d6b714431afd305ecf"}
Sep 16 18:46:41.609135 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:41.609095    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm" event={"ID":"49729b4e-8f6c-4b46-8cb0-8cae82929ab5","Type":"ContainerStarted","Data":"fbb772ef08c854bcfdc1504567af638e4fb9134f6426b9b1ed0d2d588b7ac9e7"}
Sep 16 18:46:41.609285 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:41.609157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm" event={"ID":"49729b4e-8f6c-4b46-8cb0-8cae82929ab5","Type":"ContainerStarted","Data":"04482cd637b3692bb3dfa396d4f9bb02dba99e10e4ce612cfb0d48df87c91b01"}
Sep 16 18:46:41.627148 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:41.627085    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-w9kfm" podStartSLOduration=2.067457479 podStartE2EDuration="3.627062837s" totalImagesPullingTime="1.559605358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:41.626185719 +0000 UTC m=+109.073280270" watchObservedRunningTime="2026-09-16 18:46:41.627062837 +0000 UTC m=+109.074157369"
Sep 16 18:46:43.935497 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.931035    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"]
Sep 16 18:46:43.940646 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.940617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:43.943270 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.943163    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ltmxv\""
Sep 16 18:46:43.943421 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.943399    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:46:43.943743 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.943698    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:46:43.965127 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.963656    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j2dxr"]
Sep 16 18:46:43.983802 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.983772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"]
Sep 16 18:46:43.984100 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.984087    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:43.988746 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.987605    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:46:43.988746 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.987901    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:46:43.988746 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.988545    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gf67p\""
Sep 16 18:46:43.991499 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.991196    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-tls\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5400a229-131f-4266-b766-b79aa5e298bc-metrics-client-ca\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b77a2558-789a-4ceb-a510-2bd83ec40efa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b77a2558-789a-4ceb-a510-2bd83ec40efa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b77a2558-789a-4ceb-a510-2bd83ec40efa-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2mxz\" (UniqueName: \"kubernetes.io/projected/5400a229-131f-4266-b766-b79aa5e298bc-kube-api-access-n2mxz\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-textfile\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvjc7\" (UniqueName: \"kubernetes.io/projected/b77a2558-789a-4ceb-a510-2bd83ec40efa-kube-api-access-cvjc7\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-wtmp\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5400a229-131f-4266-b766-b79aa5e298bc-root\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:43.996396 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:43.995923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5400a229-131f-4266-b766-b79aa5e298bc-sys\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.096763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvjc7\" (UniqueName: \"kubernetes.io/projected/b77a2558-789a-4ceb-a510-2bd83ec40efa-kube-api-access-cvjc7\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.096810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-wtmp\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.096842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5400a229-131f-4266-b766-b79aa5e298bc-root\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.096874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5400a229-131f-4266-b766-b79aa5e298bc-sys\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.096912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-tls\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.096943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5400a229-131f-4266-b766-b79aa5e298bc-metrics-client-ca\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.096970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b77a2558-789a-4ceb-a510-2bd83ec40efa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b77a2558-789a-4ceb-a510-2bd83ec40efa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b77a2558-789a-4ceb-a510-2bd83ec40efa-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2mxz\" (UniqueName: \"kubernetes.io/projected/5400a229-131f-4266-b766-b79aa5e298bc-kube-api-access-n2mxz\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.097241 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-textfile\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.098003 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5400a229-131f-4266-b766-b79aa5e298bc-metrics-client-ca\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.098003 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-wtmp\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.098003 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5400a229-131f-4266-b766-b79aa5e298bc-root\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.098003 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.097939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5400a229-131f-4266-b766-b79aa5e298bc-sys\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.098198 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:44.098034    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:46:44.098198 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:44.098098    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-tls podName:5400a229-131f-4266-b766-b79aa5e298bc nodeName:}" failed. No retries permitted until 2026-09-16 18:46:44.598080252 +0000 UTC m=+112.045174768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-tls") pod "node-exporter-j2dxr" (UID: "5400a229-131f-4266-b766-b79aa5e298bc") : secret "node-exporter-tls" not found
Sep 16 18:46:44.099191 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:44.098391    2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 16 18:46:44.099191 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:44.098459    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b77a2558-789a-4ceb-a510-2bd83ec40efa-openshift-state-metrics-tls podName:b77a2558-789a-4ceb-a510-2bd83ec40efa nodeName:}" failed. No retries permitted until 2026-09-16 18:46:44.598444158 +0000 UTC m=+112.045538680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b77a2558-789a-4ceb-a510-2bd83ec40efa-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-rgdxh" (UID: "b77a2558-789a-4ceb-a510-2bd83ec40efa") : secret "openshift-state-metrics-tls" not found
Sep 16 18:46:44.099191 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.098931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.099191 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.099159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-textfile\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.099447 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.099272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b77a2558-789a-4ceb-a510-2bd83ec40efa-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.101875 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.101814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b77a2558-789a-4ceb-a510-2bd83ec40efa-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.109246 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.109079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvjc7\" (UniqueName: \"kubernetes.io/projected/b77a2558-789a-4ceb-a510-2bd83ec40efa-kube-api-access-cvjc7\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.109775 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.109696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.109775 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.109728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2mxz\" (UniqueName: \"kubernetes.io/projected/5400a229-131f-4266-b766-b79aa5e298bc-kube-api-access-n2mxz\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.600158 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.600118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-tls\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.600387 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.600179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b77a2558-789a-4ceb-a510-2bd83ec40efa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.602929 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.602898    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5400a229-131f-4266-b766-b79aa5e298bc-node-exporter-tls\") pod \"node-exporter-j2dxr\" (UID: \"5400a229-131f-4266-b766-b79aa5e298bc\") " pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:44.603065 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.603001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b77a2558-789a-4ceb-a510-2bd83ec40efa-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-rgdxh\" (UID: \"b77a2558-789a-4ceb-a510-2bd83ec40efa\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.628806 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.628771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549d46475c-z29xr" event={"ID":"f38c5fef-047e-45b3-bafe-feca63d7b4c9","Type":"ContainerStarted","Data":"6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9"}
Sep 16 18:46:44.679879 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.677264    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-549d46475c-z29xr" podStartSLOduration=2.340102257 podStartE2EDuration="5.677239598s" totalImagesPullingTime="3.337137341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:44.676072435 +0000 UTC m=+112.123166991" watchObservedRunningTime="2026-09-16 18:46:44.677239598 +0000 UTC m=+112.124334130"
Sep 16 18:46:44.854902 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.854796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"
Sep 16 18:46:44.899986 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:44.899946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2dxr"
Sep 16 18:46:45.144556 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.144381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh"]
Sep 16 18:46:45.147540 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:45.147507    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77a2558_789a_4ceb_a510_2bd83ec40efa.slice/crio-dbaa674542e4841b6e68ad28b7f7d15442b47008983696c710b7df679c791675 WatchSource:0}: Error finding container dbaa674542e4841b6e68ad28b7f7d15442b47008983696c710b7df679c791675: Status 404 returned error can't find the container with id dbaa674542e4841b6e68ad28b7f7d15442b47008983696c710b7df679c791675
Sep 16 18:46:45.635576 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.635090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2dxr" event={"ID":"5400a229-131f-4266-b766-b79aa5e298bc","Type":"ContainerStarted","Data":"3d46a859ba190c56753202b0165d99725efbfae22beda1328e3035e69a0dd833"}
Sep 16 18:46:45.638449 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.638412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh" event={"ID":"b77a2558-789a-4ceb-a510-2bd83ec40efa","Type":"ContainerStarted","Data":"83f2224e4de1f6152e1917e0a9722dbe951f3c9b34d21e8f687f15657c4b6de4"}
Sep 16 18:46:45.638611 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.638498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh" event={"ID":"b77a2558-789a-4ceb-a510-2bd83ec40efa","Type":"ContainerStarted","Data":"5d74f19e5fe44f34dc7e22f3128bf34a2ec8c507b587d2972ab731017e90163c"}
Sep 16 18:46:45.638611 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.638528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh" event={"ID":"b77a2558-789a-4ceb-a510-2bd83ec40efa","Type":"ContainerStarted","Data":"dbaa674542e4841b6e68ad28b7f7d15442b47008983696c710b7df679c791675"}
Sep 16 18:46:45.902436 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.902361    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"]
Sep 16 18:46:45.907386 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.907360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:45.910354 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.910221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:46:45.910354 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.910221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:46:45.910354 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.910276    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:46:45.910718 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.910688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-277wg\""
Sep 16 18:46:45.910818 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.910798    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-42s5qdv16vuu7\""
Sep 16 18:46:45.910883 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.910824    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:46:45.910883 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.910842    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:46:45.910984 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.910922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:45.910984 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.910971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:45.911073 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.911007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-grpc-tls\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:45.911130 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.911091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/796a1899-b668-4af7-a6b1-d90d63d3c998-metrics-client-ca\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:45.911184 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.911138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-tls\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:45.911230 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.911184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghcr\" (UniqueName: \"kubernetes.io/projected/796a1899-b668-4af7-a6b1-d90d63d3c998-kube-api-access-5ghcr\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:45.911281 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.911260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:45.911326 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.911290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:45.918094 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:45.918070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"]
Sep 16 18:46:46.012211 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.012168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.012402 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.012223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.012402 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.012260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.012402 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.012283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-grpc-tls\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.012402 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.012327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/796a1899-b668-4af7-a6b1-d90d63d3c998-metrics-client-ca\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.012402 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.012366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-tls\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.012723 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.012417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghcr\" (UniqueName: \"kubernetes.io/projected/796a1899-b668-4af7-a6b1-d90d63d3c998-kube-api-access-5ghcr\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.012723 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.012464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.013271 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.013210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/796a1899-b668-4af7-a6b1-d90d63d3c998-metrics-client-ca\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.015433 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.015405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.015841 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.015817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.015938 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.015859    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-tls\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.016083 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.016059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.016610 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.016586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-grpc-tls\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.016771 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.016749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/796a1899-b668-4af7-a6b1-d90d63d3c998-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.022239 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.022215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghcr\" (UniqueName: \"kubernetes.io/projected/796a1899-b668-4af7-a6b1-d90d63d3c998-kube-api-access-5ghcr\") pod \"thanos-querier-5ddf59fb6d-btptq\" (UID: \"796a1899-b668-4af7-a6b1-d90d63d3c998\") " pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:46.219292 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:46.219207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:46:47.587345 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:47.587314    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h7lbj"
Sep 16 18:46:48.560023 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.559988    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"]
Sep 16 18:46:48.563876 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.563851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:48.567138 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.566927    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nbc58\""
Sep 16 18:46:48.567138 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.566956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:46:48.582394 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.582361    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"]
Sep 16 18:46:48.637247 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.637212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/050e092a-8786-47a9-ab97-58c8b3952f06-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zdc75\" (UID: \"050e092a-8786-47a9-ab97-58c8b3952f06\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:48.637680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.637281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752sd\" (UniqueName: \"kubernetes.io/projected/050e092a-8786-47a9-ab97-58c8b3952f06-kube-api-access-752sd\") pod \"monitoring-plugin-5d85778b9d-zdc75\" (UID: \"050e092a-8786-47a9-ab97-58c8b3952f06\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:48.738635 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.738598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/050e092a-8786-47a9-ab97-58c8b3952f06-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zdc75\" (UID: \"050e092a-8786-47a9-ab97-58c8b3952f06\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:48.738811 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.738662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752sd\" (UniqueName: \"kubernetes.io/projected/050e092a-8786-47a9-ab97-58c8b3952f06-kube-api-access-752sd\") pod \"monitoring-plugin-5d85778b9d-zdc75\" (UID: \"050e092a-8786-47a9-ab97-58c8b3952f06\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:48.738811 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:48.738789    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 18:46:48.738924 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:46:48.738869    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050e092a-8786-47a9-ab97-58c8b3952f06-monitoring-plugin-cert podName:050e092a-8786-47a9-ab97-58c8b3952f06 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:49.238847548 +0000 UTC m=+116.685942080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/050e092a-8786-47a9-ab97-58c8b3952f06-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-zdc75" (UID: "050e092a-8786-47a9-ab97-58c8b3952f06") : secret "monitoring-plugin-cert" not found
Sep 16 18:46:48.748752 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:48.748722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-752sd\" (UniqueName: \"kubernetes.io/projected/050e092a-8786-47a9-ab97-58c8b3952f06-kube-api-access-752sd\") pod \"monitoring-plugin-5d85778b9d-zdc75\" (UID: \"050e092a-8786-47a9-ab97-58c8b3952f06\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:49.242036 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:49.241991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/050e092a-8786-47a9-ab97-58c8b3952f06-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zdc75\" (UID: \"050e092a-8786-47a9-ab97-58c8b3952f06\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:49.244701 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:49.244670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/050e092a-8786-47a9-ab97-58c8b3952f06-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zdc75\" (UID: \"050e092a-8786-47a9-ab97-58c8b3952f06\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:49.475859 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:49.475815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:50.104916 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.104874    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:50.109329 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.109304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.115179 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.115140    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:46:50.115179 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.115165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:46:50.115388 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.115290    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:46:50.115710 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.115691    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:46:50.115803 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.115691    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:46:50.115871 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.115817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:46:50.116113 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.115959    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:46:50.118372 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.116326    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:46:50.118372 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.116906    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:46:50.118372 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.117090    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4gst4\""
Sep 16 18:46:50.118372 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.117261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:46:50.118372 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.117431    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:46:50.118372 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.118154    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:46:50.118758 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.118518    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9joudqapukc11\""
Sep 16 18:46:50.118758 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.118745    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:46:50.121584 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.121517    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:46:50.130349 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.130325    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:46:50.140495 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.140448    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:50.151657 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.151825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.151825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.151825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.151825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152045 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152045 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152045 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152045 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152045 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.151986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152294 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152294 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152294 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152294 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4q7\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-kube-api-access-4w4q7\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152294 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152294 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152294 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152294 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152666 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.152666 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.152342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.198094 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.197726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:50.198094 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.198053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:50.203145 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.203120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:50.253265 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253265 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253469 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253469 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253469 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253469 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253469 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253700 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4q7\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-kube-api-access-4w4q7\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253700 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253700 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253700 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253700 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253700 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.253700 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.254008 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.254008 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.254008 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.254008 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.254008 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.254008 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.253939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.254296 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.254138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.254296 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.254194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.254853 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.254814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.257063 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.255370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.257063 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.256375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.257063 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.256539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.257063 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.257010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.257352 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.257319    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.257767 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.257740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.257982 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.257915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.257982 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.257914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.259834 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.259778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.260098 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.260025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.260098 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.260027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.260725 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.260528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.260725 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.260681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.261118 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.261095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.261257 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.261236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.261699 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.261680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.270782 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.270757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4q7\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-kube-api-access-4w4q7\") pod \"prometheus-k8s-0\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.426411 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.426308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:50.668777 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:50.668743    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:46:52.817782 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:52.817612    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"]
Sep 16 18:46:52.821529 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:52.821471    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050e092a_8786_47a9_ab97_58c8b3952f06.slice/crio-9d5c9d1678cfa85b9e3d2887d49e55de22960c49e281a4c5def27f108e8a1ddf WatchSource:0}: Error finding container 9d5c9d1678cfa85b9e3d2887d49e55de22960c49e281a4c5def27f108e8a1ddf: Status 404 returned error can't find the container with id 9d5c9d1678cfa85b9e3d2887d49e55de22960c49e281a4c5def27f108e8a1ddf
Sep 16 18:46:52.828050 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:52.827402    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"]
Sep 16 18:46:52.830751 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:52.830659    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:52.832925 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:52.832887    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796a1899_b668_4af7_a6b1_d90d63d3c998.slice/crio-157a9d0f881c3fdb3729ff139c5a2cc801df959e9367323f08152defd709c8ce WatchSource:0}: Error finding container 157a9d0f881c3fdb3729ff139c5a2cc801df959e9367323f08152defd709c8ce: Status 404 returned error can't find the container with id 157a9d0f881c3fdb3729ff139c5a2cc801df959e9367323f08152defd709c8ce
Sep 16 18:46:52.840894 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:52.840384    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dafa50f_d9a5_4c6d_83ef_c8064a55a01f.slice/crio-0b5ef0b3051e106928f6af7f814791561e9a31ca25825c0b9ca0eb13aac8ebad WatchSource:0}: Error finding container 0b5ef0b3051e106928f6af7f814791561e9a31ca25825c0b9ca0eb13aac8ebad: Status 404 returned error can't find the container with id 0b5ef0b3051e106928f6af7f814791561e9a31ca25825c0b9ca0eb13aac8ebad
Sep 16 18:46:53.679973 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:53.679910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerStarted","Data":"0b5ef0b3051e106928f6af7f814791561e9a31ca25825c0b9ca0eb13aac8ebad"}
Sep 16 18:46:53.682505 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:53.681906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75" event={"ID":"050e092a-8786-47a9-ab97-58c8b3952f06","Type":"ContainerStarted","Data":"9d5c9d1678cfa85b9e3d2887d49e55de22960c49e281a4c5def27f108e8a1ddf"}
Sep 16 18:46:53.685417 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:53.685150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-gl9hm" event={"ID":"2ca91436-3300-447c-a199-5e12d7b3f511","Type":"ContainerStarted","Data":"a57288d3672fae4f8341da55e8f414c1886fe7dfd8b12491e3393a67a2c8ac46"}
Sep 16 18:46:53.685792 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:53.685735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-gl9hm"
Sep 16 18:46:53.687896 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:53.687818    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq" event={"ID":"796a1899-b668-4af7-a6b1-d90d63d3c998","Type":"ContainerStarted","Data":"157a9d0f881c3fdb3729ff139c5a2cc801df959e9367323f08152defd709c8ce"}
Sep 16 18:46:53.702780 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:53.702729    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-gl9hm"
Sep 16 18:46:53.738671 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:53.738542    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-gl9hm" podStartSLOduration=2.4043062920000002 podStartE2EDuration="19.738522041s" totalImagesPullingTime="17.334215749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:53.714829567 +0000 UTC m=+121.161924113" watchObservedRunningTime="2026-09-16 18:46:53.738522041 +0000 UTC m=+121.185616570"
Sep 16 18:46:54.695834 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:54.695789    2578 generic.go:356] "Generic (PLEG): container finished" podID="5400a229-131f-4266-b766-b79aa5e298bc" containerID="94e5e41c1d4fd438ec3469d04861a17fed5801c6d779881bc6ae628abb696a9c" exitCode=0
Sep 16 18:46:54.696329 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:54.695893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2dxr" event={"ID":"5400a229-131f-4266-b766-b79aa5e298bc","Type":"ContainerDied","Data":"94e5e41c1d4fd438ec3469d04861a17fed5801c6d779881bc6ae628abb696a9c"}
Sep 16 18:46:54.699390 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:54.699353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh" event={"ID":"b77a2558-789a-4ceb-a510-2bd83ec40efa","Type":"ContainerStarted","Data":"fe163f39f14d6701f36c13b7eca9fdf64e219a7e62f6b70d22ccb22ddf709639"}
Sep 16 18:46:54.756494 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:54.756371    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-rgdxh" podStartSLOduration=3.305565426 podStartE2EDuration="11.756351626s" totalImagesPullingTime="8.4507862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:54.75498924 +0000 UTC m=+122.202083771" watchObservedRunningTime="2026-09-16 18:46:54.756351626 +0000 UTC m=+122.203446156"
Sep 16 18:46:57.715623 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.715581    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerID="615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c" exitCode=0
Sep 16 18:46:57.716138 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.715629    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerDied","Data":"615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c"}
Sep 16 18:46:57.717666 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.717638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75" event={"ID":"050e092a-8786-47a9-ab97-58c8b3952f06","Type":"ContainerStarted","Data":"97fe9db41ead6085d7984f0e93d05e1a7cb06386f3931778fbeb07d84cbd9cd0"}
Sep 16 18:46:57.718141 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.718116    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:57.720496 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.720449    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2dxr" event={"ID":"5400a229-131f-4266-b766-b79aa5e298bc","Type":"ContainerStarted","Data":"f8c4e737f822f6c00109dbbe1b9fa06dd208e643a064abd8d1992d5651bf658b"}
Sep 16 18:46:57.720599 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.720529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2dxr" event={"ID":"5400a229-131f-4266-b766-b79aa5e298bc","Type":"ContainerStarted","Data":"cfd182148eb9be4b33ca71b7d6d28b95075f0807c71c303c3305db29c46bd6a4"}
Sep 16 18:46:57.723870 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.723849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq" event={"ID":"796a1899-b668-4af7-a6b1-d90d63d3c998","Type":"ContainerStarted","Data":"78632f503b8da06c5621de6ba557f450668c591ad66535017f37f1b1701da819"}
Sep 16 18:46:57.723973 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.723914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq" event={"ID":"796a1899-b668-4af7-a6b1-d90d63d3c998","Type":"ContainerStarted","Data":"1cb743ab2311085841073a8ceb932163d46a6237d26e53ba90916c65ad693044"}
Sep 16 18:46:57.723973 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.723959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq" event={"ID":"796a1899-b668-4af7-a6b1-d90d63d3c998","Type":"ContainerStarted","Data":"446e4eb823263ae11569c58c8d1918d50a3fc23bcfe2dc6cf357fcda0392605c"}
Sep 16 18:46:57.724613 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.724591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75"
Sep 16 18:46:57.769594 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.769536    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j2dxr" podStartSLOduration=5.821460575 podStartE2EDuration="14.769512573s" totalImagesPullingTime="8.822859443s" totalInitContainerRuntime="125.192555ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:57.768462967 +0000 UTC m=+125.215557518" watchObservedRunningTime="2026-09-16 18:46:57.769512573 +0000 UTC m=+125.216607106"
Sep 16 18:46:57.786533 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:57.786469    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zdc75" podStartSLOduration=5.77129705 podStartE2EDuration="9.786449643s" totalImagesPullingTime="4.015152593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:57.786200717 +0000 UTC m=+125.233295270" watchObservedRunningTime="2026-09-16 18:46:57.786449643 +0000 UTC m=+125.233544173"
Sep 16 18:46:58.538735 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.538586    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69467445df-hwcxv"]
Sep 16 18:46:58.560886 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.560841    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69467445df-hwcxv"]
Sep 16 18:46:58.561056 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.561010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.572001 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.571976    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:46:58.647014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.646599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-trusted-ca-bundle\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.647014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.646651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-oauth-serving-cert\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.647014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.646696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-serving-cert\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.647014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.646731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-console-config\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.647014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.646766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-service-ca\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.647014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.646789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp2xb\" (UniqueName: \"kubernetes.io/projected/19009e13-f275-4830-b95d-4f4a98fdfec2-kube-api-access-sp2xb\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.647014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.646844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-oauth-config\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.748213 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.748173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-trusted-ca-bundle\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.748225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-oauth-serving-cert\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.748282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-serving-cert\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.748316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-console-config\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.748366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-service-ca\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.748395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2xb\" (UniqueName: \"kubernetes.io/projected/19009e13-f275-4830-b95d-4f4a98fdfec2-kube-api-access-sp2xb\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.748452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-oauth-config\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.749164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-trusted-ca-bundle\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.749516    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-console-config\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.749749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-service-ca\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.750327 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.749793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-oauth-serving-cert\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.751568 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.751542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-oauth-config\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.752732 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.752710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-serving-cert\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.759200 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.759173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2xb\" (UniqueName: \"kubernetes.io/projected/19009e13-f275-4830-b95d-4f4a98fdfec2-kube-api-access-sp2xb\") pod \"console-69467445df-hwcxv\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") " pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:58.879212 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:58.879133    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:46:59.229190 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:59.229111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69467445df-hwcxv"]
Sep 16 18:46:59.237201 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:46:59.237168    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19009e13_f275_4830_b95d_4f4a98fdfec2.slice/crio-6cfe9845f57cc2b386a26de73b15fb84b2148162fff4c3317508222df94f6414 WatchSource:0}: Error finding container 6cfe9845f57cc2b386a26de73b15fb84b2148162fff4c3317508222df94f6414: Status 404 returned error can't find the container with id 6cfe9845f57cc2b386a26de73b15fb84b2148162fff4c3317508222df94f6414
Sep 16 18:46:59.738530 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:59.737972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69467445df-hwcxv" event={"ID":"19009e13-f275-4830-b95d-4f4a98fdfec2","Type":"ContainerStarted","Data":"ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7"}
Sep 16 18:46:59.738530 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:59.738035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69467445df-hwcxv" event={"ID":"19009e13-f275-4830-b95d-4f4a98fdfec2","Type":"ContainerStarted","Data":"6cfe9845f57cc2b386a26de73b15fb84b2148162fff4c3317508222df94f6414"}
Sep 16 18:46:59.741970 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:59.741935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq" event={"ID":"796a1899-b668-4af7-a6b1-d90d63d3c998","Type":"ContainerStarted","Data":"aeef19de58d22eecc12c38c94f8f0163d401699eb887397550bb6ee2fa1816ee"}
Sep 16 18:46:59.742123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:59.742012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq" event={"ID":"796a1899-b668-4af7-a6b1-d90d63d3c998","Type":"ContainerStarted","Data":"884d94092dd4b00e1eb8643788a2bc7618945cab22d192e6a21e5c0aef9a54f6"}
Sep 16 18:46:59.742123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:59.742053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq" event={"ID":"796a1899-b668-4af7-a6b1-d90d63d3c998","Type":"ContainerStarted","Data":"32ee253d06cc28747eb6668802c311823bb497a08a5a2ee86d11bcdfcc00555a"}
Sep 16 18:46:59.761244 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:59.761185    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69467445df-hwcxv" podStartSLOduration=1.761163828 podStartE2EDuration="1.761163828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:59.759568152 +0000 UTC m=+127.206662707" watchObservedRunningTime="2026-09-16 18:46:59.761163828 +0000 UTC m=+127.208258361"
Sep 16 18:46:59.791081 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:46:59.791020    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq" podStartSLOduration=9.15065721 podStartE2EDuration="14.790997794s" totalImagesPullingTime="5.640340584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:59.79080226 +0000 UTC m=+127.237896810" watchObservedRunningTime="2026-09-16 18:46:59.790997794 +0000 UTC m=+127.238092371"
Sep 16 18:47:00.748640 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:00.748609    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:47:01.884172 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:01.884136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:47:01.886836 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:01.886808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd44d99b-8aab-441b-a376-af9a0d55f130-metrics-certs\") pod \"network-metrics-daemon-6fpgx\" (UID: \"dd44d99b-8aab-441b-a376-af9a0d55f130\") " pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:47:01.893010 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:01.892983    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pq9cp\""
Sep 16 18:47:01.900307 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:01.900280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fpgx"
Sep 16 18:47:02.229312 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:02.229216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6fpgx"]
Sep 16 18:47:02.231712 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:47:02.231684    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd44d99b_8aab_441b_a376_af9a0d55f130.slice/crio-7635f65fa8dfdc2df1d8f5855ddd412eb736849e0cec5d24e6c77227a9cc38be WatchSource:0}: Error finding container 7635f65fa8dfdc2df1d8f5855ddd412eb736849e0cec5d24e6c77227a9cc38be: Status 404 returned error can't find the container with id 7635f65fa8dfdc2df1d8f5855ddd412eb736849e0cec5d24e6c77227a9cc38be
Sep 16 18:47:02.759022 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:02.758983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fpgx" event={"ID":"dd44d99b-8aab-441b-a376-af9a0d55f130","Type":"ContainerStarted","Data":"7635f65fa8dfdc2df1d8f5855ddd412eb736849e0cec5d24e6c77227a9cc38be"}
Sep 16 18:47:02.763194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:02.763161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerStarted","Data":"e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129"}
Sep 16 18:47:02.763350 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:02.763228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerStarted","Data":"422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181"}
Sep 16 18:47:02.763350 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:02.763258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerStarted","Data":"addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a"}
Sep 16 18:47:02.763350 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:02.763292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerStarted","Data":"281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2"}
Sep 16 18:47:02.763350 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:02.763318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerStarted","Data":"85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65"}
Sep 16 18:47:02.763350 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:02.763348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerStarted","Data":"26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a"}
Sep 16 18:47:02.803649 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:02.803580    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.776738888 podStartE2EDuration="12.803560918s" totalImagesPullingTime="7.99321299s" totalInitContainerRuntime="33.60904ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:02.800526473 +0000 UTC m=+130.247621004" watchObservedRunningTime="2026-09-16 18:47:02.803560918 +0000 UTC m=+130.250655448"
Sep 16 18:47:03.774548 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:03.774502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fpgx" event={"ID":"dd44d99b-8aab-441b-a376-af9a0d55f130","Type":"ContainerStarted","Data":"6627e33cba4cf3736f6b33a8d16d99bfac316d56a1fd44a3484c441dba76e63a"}
Sep 16 18:47:04.781075 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:04.781036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fpgx" event={"ID":"dd44d99b-8aab-441b-a376-af9a0d55f130","Type":"ContainerStarted","Data":"2e9568d54c6a2bbb4a7ebec26f829719dee27b8784d6f9e82f0740107ace6b7c"}
Sep 16 18:47:04.799612 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:04.799557    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6fpgx" podStartSLOduration=130.500021384 podStartE2EDuration="2m11.799539741s" totalImagesPullingTime="1.299518357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:04.798596292 +0000 UTC m=+132.245690850" watchObservedRunningTime="2026-09-16 18:47:04.799539741 +0000 UTC m=+132.246634270"
Sep 16 18:47:05.426752 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:05.426718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:06.758917 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:06.758890    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5ddf59fb6d-btptq"
Sep 16 18:47:08.879616 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:08.879572    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:47:08.880160 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:08.879648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:47:08.884520 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:08.884498    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:47:09.807471 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:09.807442    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:47:09.863178 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:09.863136    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549d46475c-z29xr"]
Sep 16 18:47:19.845243 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:19.845210    2578 generic.go:356] "Generic (PLEG): container finished" podID="d02c7f84-f3da-432a-9cc3-2588b5256190" containerID="53807f0e6e1a5e73b2664d3f33d55660bc26181a42efe2cd52eb2dfe4c031d63" exitCode=0
Sep 16 18:47:19.845804 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:19.845283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7" event={"ID":"d02c7f84-f3da-432a-9cc3-2588b5256190","Type":"ContainerDied","Data":"53807f0e6e1a5e73b2664d3f33d55660bc26181a42efe2cd52eb2dfe4c031d63"}
Sep 16 18:47:19.845804 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:19.845611    2578 scope.go:117] "RemoveContainer" containerID="53807f0e6e1a5e73b2664d3f33d55660bc26181a42efe2cd52eb2dfe4c031d63"
Sep 16 18:47:20.850845 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:20.850809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zgkp7" event={"ID":"d02c7f84-f3da-432a-9cc3-2588b5256190","Type":"ContainerStarted","Data":"944c0c0ed6b7c580495d5d7814875aa81e92c94b5d96e2e60006b32fa9c791f2"}
Sep 16 18:47:34.882667 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:34.882603    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-549d46475c-z29xr" podUID="f38c5fef-047e-45b3-bafe-feca63d7b4c9" containerName="console" containerID="cri-o://6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9" gracePeriod=15
Sep 16 18:47:35.149268 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.149200    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:47:35.176807 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.176772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qc2c\" (UniqueName: \"kubernetes.io/projected/f38c5fef-047e-45b3-bafe-feca63d7b4c9-kube-api-access-5qc2c\") pod \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") "
Sep 16 18:47:35.176963 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.176829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-service-ca\") pod \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") "
Sep 16 18:47:35.176963 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.176911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-serving-cert\") pod \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") "
Sep 16 18:47:35.176963 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.176937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-oauth-serving-cert\") pod \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") "
Sep 16 18:47:35.177089 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.177009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-config\") pod \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") "
Sep 16 18:47:35.177089 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.177067    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-oauth-config\") pod \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\" (UID: \"f38c5fef-047e-45b3-bafe-feca63d7b4c9\") "
Sep 16 18:47:35.177383 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.177346    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-service-ca" (OuterVolumeSpecName: "service-ca") pod "f38c5fef-047e-45b3-bafe-feca63d7b4c9" (UID: "f38c5fef-047e-45b3-bafe-feca63d7b4c9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:35.177528 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.177433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f38c5fef-047e-45b3-bafe-feca63d7b4c9" (UID: "f38c5fef-047e-45b3-bafe-feca63d7b4c9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:35.177528 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.177506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-config" (OuterVolumeSpecName: "console-config") pod "f38c5fef-047e-45b3-bafe-feca63d7b4c9" (UID: "f38c5fef-047e-45b3-bafe-feca63d7b4c9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:35.179214 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.179192    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f38c5fef-047e-45b3-bafe-feca63d7b4c9" (UID: "f38c5fef-047e-45b3-bafe-feca63d7b4c9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:35.179331 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.179247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f38c5fef-047e-45b3-bafe-feca63d7b4c9" (UID: "f38c5fef-047e-45b3-bafe-feca63d7b4c9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:35.179331 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.179277    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38c5fef-047e-45b3-bafe-feca63d7b4c9-kube-api-access-5qc2c" (OuterVolumeSpecName: "kube-api-access-5qc2c") pod "f38c5fef-047e-45b3-bafe-feca63d7b4c9" (UID: "f38c5fef-047e-45b3-bafe-feca63d7b4c9"). InnerVolumeSpecName "kube-api-access-5qc2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:35.278270 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.278227    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-serving-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:47:35.278270 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.278259    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-oauth-serving-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:47:35.278270 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.278271    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-config\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:47:35.278270 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.278281    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f38c5fef-047e-45b3-bafe-feca63d7b4c9-console-oauth-config\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:47:35.278611 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.278290    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qc2c\" (UniqueName: \"kubernetes.io/projected/f38c5fef-047e-45b3-bafe-feca63d7b4c9-kube-api-access-5qc2c\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:47:35.278611 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.278298    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f38c5fef-047e-45b3-bafe-feca63d7b4c9-service-ca\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:47:35.914897 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.914864    2578 generic.go:356] "Generic (PLEG): container finished" podID="f38c5fef-047e-45b3-bafe-feca63d7b4c9" containerID="6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9" exitCode=0
Sep 16 18:47:35.915395 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.914923    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549d46475c-z29xr"
Sep 16 18:47:35.915395 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.914961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549d46475c-z29xr" event={"ID":"f38c5fef-047e-45b3-bafe-feca63d7b4c9","Type":"ContainerDied","Data":"6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9"}
Sep 16 18:47:35.915395 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.915012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549d46475c-z29xr" event={"ID":"f38c5fef-047e-45b3-bafe-feca63d7b4c9","Type":"ContainerDied","Data":"168ec95417b8a1ec3dd1ac480db44295a80e989c118a74d6b714431afd305ecf"}
Sep 16 18:47:35.915395 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.915036    2578 scope.go:117] "RemoveContainer" containerID="6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9"
Sep 16 18:47:35.924025 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.924007    2578 scope.go:117] "RemoveContainer" containerID="6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9"
Sep 16 18:47:35.924314 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:47:35.924295    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9\": container with ID starting with 6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9 not found: ID does not exist" containerID="6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9"
Sep 16 18:47:35.924359 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.924322    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9"} err="failed to get container status \"6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9\": rpc error: code = NotFound desc = could not find container \"6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9\": container with ID starting with 6fafa27be8d60554a544c8860c390cd343c05b1813df598e8f86ad835208d8a9 not found: ID does not exist"
Sep 16 18:47:35.937020 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.936978    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549d46475c-z29xr"]
Sep 16 18:47:35.940962 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:35.940938    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-549d46475c-z29xr"]
Sep 16 18:47:37.066253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:37.066219    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f38c5fef-047e-45b3-bafe-feca63d7b4c9" path="/var/lib/kubelet/pods/f38c5fef-047e-45b3-bafe-feca63d7b4c9/volumes"
Sep 16 18:47:50.426710 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:50.426606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:50.442525 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:50.442500    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:50.994243 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:47:50.994211    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:09.102668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:09.102630    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:48:09.103750 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:09.103712    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="prometheus" containerID="cri-o://26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a" gracePeriod=600
Sep 16 18:48:09.104199 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:09.104146    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy" containerID="cri-o://422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181" gracePeriod=600
Sep 16 18:48:09.104305 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:09.103916    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129" gracePeriod=600
Sep 16 18:48:09.104305 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:09.104120    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="config-reloader" containerID="cri-o://85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65" gracePeriod=600
Sep 16 18:48:09.104305 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:09.104146    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="thanos-sidecar" containerID="cri-o://281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2" gracePeriod=600
Sep 16 18:48:09.104473 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:09.104151    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy-web" containerID="cri-o://addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a" gracePeriod=600
Sep 16 18:48:10.064123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064090    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerID="e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129" exitCode=0
Sep 16 18:48:10.064123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064114    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerID="422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181" exitCode=0
Sep 16 18:48:10.064123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064122    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerID="281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2" exitCode=0
Sep 16 18:48:10.064123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064128    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerID="85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65" exitCode=0
Sep 16 18:48:10.064123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064133    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerID="26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a" exitCode=0
Sep 16 18:48:10.064410 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerDied","Data":"e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129"}
Sep 16 18:48:10.064410 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerDied","Data":"422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181"}
Sep 16 18:48:10.064410 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerDied","Data":"281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2"}
Sep 16 18:48:10.064410 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerDied","Data":"85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65"}
Sep 16 18:48:10.064410 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.064288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerDied","Data":"26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a"}
Sep 16 18:48:10.354811 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.354777    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:10.395704 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395673    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-2\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.395895 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config-out\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.395895 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.395895 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395773    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-grpc-tls\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.395895 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395799    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-trusted-ca-bundle\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.395895 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-kubelet-serving-ca-bundle\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.395895 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395857    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-db\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-thanos-prometheus-http-client-file\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-0\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.395974    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-web-config\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396004    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-metrics-client-ca\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w4q7\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-kube-api-access-4w4q7\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-1\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:10.396194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396093    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-tls\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396194 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396166    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-metrics-client-certs\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396210    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396208    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:10.396668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-serving-certs-ca-bundle\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396304    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-kube-rbac-proxy\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396328    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-tls-assets\") pod \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\" (UID: \"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f\") "
Sep 16 18:48:10.396668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:10.396668 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396669    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.397068 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396689    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.397068 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396705    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.397068 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.396904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:10.397242 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.397218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:10.398831 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.398801    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:10.399266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.399175    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:10.399266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.399238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config-out" (OuterVolumeSpecName: "config-out") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:10.399617 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.399532    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-kube-api-access-4w4q7" (OuterVolumeSpecName: "kube-api-access-4w4q7") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "kube-api-access-4w4q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:10.399617 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.399592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:10.399925 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.399828    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:10.399925 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.399904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config" (OuterVolumeSpecName: "config") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:10.401054 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.401026    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:10.401147 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.401074    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:10.401147 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.401112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:10.401147 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.401131    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:10.401543 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.401516    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:10.401646 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.401557    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:10.402258 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.402240    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:10.412947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.412903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-web-config" (OuterVolumeSpecName: "web-config") pod "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" (UID: "9dafa50f-d9a5-4c6d-83ef-c8064a55a01f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:10.497720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497685    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4q7\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-kube-api-access-4w4q7\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497713    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497725    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497733    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-metrics-client-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497743    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497754    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497764    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497773    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-kube-rbac-proxy\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497781    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-tls-assets\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497788    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-config-out\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497796    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497804    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-secret-grpc-tls\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497812    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-db\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497820    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497830    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497839    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-web-config\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:10.497947 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:10.497849    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f-configmap-metrics-client-ca\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:48:11.071414 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.071135    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerID="addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a" exitCode=0
Sep 16 18:48:11.071414 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.071185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerDied","Data":"addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a"}
Sep 16 18:48:11.071414 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.071247    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.071414 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.071262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dafa50f-d9a5-4c6d-83ef-c8064a55a01f","Type":"ContainerDied","Data":"0b5ef0b3051e106928f6af7f814791561e9a31ca25825c0b9ca0eb13aac8ebad"}
Sep 16 18:48:11.071414 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.071309    2578 scope.go:117] "RemoveContainer" containerID="e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129"
Sep 16 18:48:11.079013 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.078993    2578 scope.go:117] "RemoveContainer" containerID="422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181"
Sep 16 18:48:11.085833 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.085814    2578 scope.go:117] "RemoveContainer" containerID="addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a"
Sep 16 18:48:11.092667 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.092647    2578 scope.go:117] "RemoveContainer" containerID="281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2"
Sep 16 18:48:11.099363 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.099340    2578 scope.go:117] "RemoveContainer" containerID="85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65"
Sep 16 18:48:11.103918 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.103538    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:48:11.110282 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.110256    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:48:11.111094 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.111077    2578 scope.go:117] "RemoveContainer" containerID="26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a"
Sep 16 18:48:11.118825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.118804    2578 scope.go:117] "RemoveContainer" containerID="615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c"
Sep 16 18:48:11.125691 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.125673    2578 scope.go:117] "RemoveContainer" containerID="e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129"
Sep 16 18:48:11.125970 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:48:11.125952    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129\": container with ID starting with e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129 not found: ID does not exist" containerID="e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129"
Sep 16 18:48:11.126023 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.125979    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129"} err="failed to get container status \"e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129\": rpc error: code = NotFound desc = could not find container \"e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129\": container with ID starting with e765d1f5d80fc85296081c0ff484a8b8fca63e54c29cf6e926fc4a2b0ec99129 not found: ID does not exist"
Sep 16 18:48:11.126023 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.125999    2578 scope.go:117] "RemoveContainer" containerID="422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181"
Sep 16 18:48:11.126227 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:48:11.126203    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181\": container with ID starting with 422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181 not found: ID does not exist" containerID="422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181"
Sep 16 18:48:11.126309 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.126236    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181"} err="failed to get container status \"422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181\": rpc error: code = NotFound desc = could not find container \"422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181\": container with ID starting with 422b563dc4292359eb27a2be26a93acc93bc3dd2373ab6365d9be8e325a1e181 not found: ID does not exist"
Sep 16 18:48:11.126309 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.126258    2578 scope.go:117] "RemoveContainer" containerID="addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a"
Sep 16 18:48:11.126592 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:48:11.126572    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a\": container with ID starting with addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a not found: ID does not exist" containerID="addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a"
Sep 16 18:48:11.126637 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.126596    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a"} err="failed to get container status \"addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a\": rpc error: code = NotFound desc = could not find container \"addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a\": container with ID starting with addcda53eb353b250cd46d95046847e5a4cce618ded907360ca4df4d5e68cb9a not found: ID does not exist"
Sep 16 18:48:11.126637 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.126608    2578 scope.go:117] "RemoveContainer" containerID="281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2"
Sep 16 18:48:11.126846 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:48:11.126827    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2\": container with ID starting with 281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2 not found: ID does not exist" containerID="281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2"
Sep 16 18:48:11.126889 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.126851    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2"} err="failed to get container status \"281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2\": rpc error: code = NotFound desc = could not find container \"281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2\": container with ID starting with 281bb4809ee0cf87a8abc5479f80f7dcfd478c663b3696d0693723ada98229a2 not found: ID does not exist"
Sep 16 18:48:11.126889 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.126865    2578 scope.go:117] "RemoveContainer" containerID="85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65"
Sep 16 18:48:11.127069 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:48:11.127054    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65\": container with ID starting with 85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65 not found: ID does not exist" containerID="85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65"
Sep 16 18:48:11.127112 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.127071    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65"} err="failed to get container status \"85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65\": rpc error: code = NotFound desc = could not find container \"85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65\": container with ID starting with 85362564ed362008fefb92c8caae71983e2dfecad0ed454151f4a822a02b4f65 not found: ID does not exist"
Sep 16 18:48:11.127112 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.127083    2578 scope.go:117] "RemoveContainer" containerID="26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a"
Sep 16 18:48:11.127286 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:48:11.127273    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a\": container with ID starting with 26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a not found: ID does not exist" containerID="26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a"
Sep 16 18:48:11.127320 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.127288    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a"} err="failed to get container status \"26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a\": rpc error: code = NotFound desc = could not find container \"26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a\": container with ID starting with 26a679aac93d49a90b727c49672b8c69a790b57fbf3779b8b2a3beef7fbcde8a not found: ID does not exist"
Sep 16 18:48:11.127320 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.127299    2578 scope.go:117] "RemoveContainer" containerID="615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c"
Sep 16 18:48:11.127560 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:48:11.127543    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c\": container with ID starting with 615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c not found: ID does not exist" containerID="615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c"
Sep 16 18:48:11.127648 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.127565    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c"} err="failed to get container status \"615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c\": rpc error: code = NotFound desc = could not find container \"615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c\": container with ID starting with 615bb0ad971a491a1480d240edf0a089b06ea1b9e1c0caee74e2619123e56c7c not found: ID does not exist"
Sep 16 18:48:11.145144 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145101    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:48:11.145470 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145453    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="thanos-sidecar"
Sep 16 18:48:11.145470 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145470    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="thanos-sidecar"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145499    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy-web"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145506    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy-web"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145515    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38c5fef-047e-45b3-bafe-feca63d7b4c9" containerName="console"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145521    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38c5fef-047e-45b3-bafe-feca63d7b4c9" containerName="console"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145529    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="prometheus"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145534    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="prometheus"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145544    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="config-reloader"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145550    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="config-reloader"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145558    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy-thanos"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145566    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy-thanos"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145575    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="init-config-reloader"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145581    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="init-config-reloader"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145590    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy"
Sep 16 18:48:11.145603 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145595    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy"
Sep 16 18:48:11.146002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145648    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38c5fef-047e-45b3-bafe-feca63d7b4c9" containerName="console"
Sep 16 18:48:11.146002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145657    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="thanos-sidecar"
Sep 16 18:48:11.146002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145664    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy"
Sep 16 18:48:11.146002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145672    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="config-reloader"
Sep 16 18:48:11.146002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145677    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy-web"
Sep 16 18:48:11.146002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145683    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="kube-rbac-proxy-thanos"
Sep 16 18:48:11.146002 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.145690    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" containerName="prometheus"
Sep 16 18:48:11.149574 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.149559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.155067 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.152674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4gst4\""
Sep 16 18:48:11.155067 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.153121    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:48:11.155341 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.155279    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:48:11.155341 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.155310    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:48:11.155518 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.155310    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:48:11.155518 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.155406    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:48:11.155631 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.155590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9joudqapukc11\""
Sep 16 18:48:11.155680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.155638    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:48:11.155680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.155661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:48:11.155680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.155671    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:48:11.156015 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.156001    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:48:11.156094 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.156079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:48:11.156520 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.156370    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:48:11.158069 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.158053    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:48:11.159297 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.159283    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:48:11.163907 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.163888    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:48:11.169561 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.169540    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:48:11.182574 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.182541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:48:11.205497 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205697 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205697 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205697 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205697 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205697 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0db0dce-59ad-48d4-97fa-d97a54af1bce-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205945 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hl98\" (UniqueName: \"kubernetes.io/projected/a0db0dce-59ad-48d4-97fa-d97a54af1bce-kube-api-access-7hl98\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205945 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205945 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205945 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205945 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-config\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205945 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.205945 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0db0dce-59ad-48d4-97fa-d97a54af1bce-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.206293 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.206293 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.205987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.206293 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.206002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.206293 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.206027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.206293 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.206048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.206293 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.206133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.206293 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.206180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307544 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307544 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0db0dce-59ad-48d4-97fa-d97a54af1bce-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hl98\" (UniqueName: \"kubernetes.io/projected/a0db0dce-59ad-48d4-97fa-d97a54af1bce-kube-api-access-7hl98\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-config\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0db0dce-59ad-48d4-97fa-d97a54af1bce-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.307849 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.308418 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.308418 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.307912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.308418 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.308320    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.308418 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.308353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.308418 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.308367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.309629 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.309120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.309629 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.309175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.309629 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.309210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.309629 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.309249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.309629 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.309542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.310244 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.309695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.310244 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.309993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.310713 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.310667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.311014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.310990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.311777 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.311754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0db0dce-59ad-48d4-97fa-d97a54af1bce-config-out\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.311944 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.311921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0db0dce-59ad-48d4-97fa-d97a54af1bce-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.312147 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.312122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.312265 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.312242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-config\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.312373 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.312306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.313248 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.313221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.313358 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.313286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a0db0dce-59ad-48d4-97fa-d97a54af1bce-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.313635 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.313613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.313737 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.313656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.313737 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.313730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.313993 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.313973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0db0dce-59ad-48d4-97fa-d97a54af1bce-web-config\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.317351 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.317332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hl98\" (UniqueName: \"kubernetes.io/projected/a0db0dce-59ad-48d4-97fa-d97a54af1bce-kube-api-access-7hl98\") pod \"prometheus-k8s-0\" (UID: \"a0db0dce-59ad-48d4-97fa-d97a54af1bce\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.460988 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.460898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:11.748248 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:11.748218    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:48:11.749678 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:48:11.749653    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0db0dce_59ad_48d4_97fa_d97a54af1bce.slice/crio-f21f5b75076b16111d2f815e5d0e114e8f73d8bfcf5afb35aec31f7d1b088a25 WatchSource:0}: Error finding container f21f5b75076b16111d2f815e5d0e114e8f73d8bfcf5afb35aec31f7d1b088a25: Status 404 returned error can't find the container with id f21f5b75076b16111d2f815e5d0e114e8f73d8bfcf5afb35aec31f7d1b088a25
Sep 16 18:48:12.078581 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:12.078539    2578 generic.go:356] "Generic (PLEG): container finished" podID="a0db0dce-59ad-48d4-97fa-d97a54af1bce" containerID="06c0da3285a6a935933b4256aff61eb83f62e813b0e6553bccbf3e100c5bc7e3" exitCode=0
Sep 16 18:48:12.078722 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:12.078592    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0db0dce-59ad-48d4-97fa-d97a54af1bce","Type":"ContainerDied","Data":"06c0da3285a6a935933b4256aff61eb83f62e813b0e6553bccbf3e100c5bc7e3"}
Sep 16 18:48:12.078722 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:12.078655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0db0dce-59ad-48d4-97fa-d97a54af1bce","Type":"ContainerStarted","Data":"f21f5b75076b16111d2f815e5d0e114e8f73d8bfcf5afb35aec31f7d1b088a25"}
Sep 16 18:48:13.063223 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:13.063187    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dafa50f-d9a5-4c6d-83ef-c8064a55a01f" path="/var/lib/kubelet/pods/9dafa50f-d9a5-4c6d-83ef-c8064a55a01f/volumes"
Sep 16 18:48:13.086416 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:13.086382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0db0dce-59ad-48d4-97fa-d97a54af1bce","Type":"ContainerStarted","Data":"b3acd9107bbcebcd7472cbb7cdaed2e30230ca99d997bf720a73700dbf8a85e2"}
Sep 16 18:48:13.086587 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:13.086448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0db0dce-59ad-48d4-97fa-d97a54af1bce","Type":"ContainerStarted","Data":"375f910be82d85747f0d6045cbb42ee1715789bd5a3bcf0383aaa332a9ef9663"}
Sep 16 18:48:13.086587 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:13.086490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0db0dce-59ad-48d4-97fa-d97a54af1bce","Type":"ContainerStarted","Data":"f5adc47f7312410bb1b4706c14684323f0b535637aba92a1ce1ba1075133c582"}
Sep 16 18:48:13.086587 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:13.086514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0db0dce-59ad-48d4-97fa-d97a54af1bce","Type":"ContainerStarted","Data":"689d8ec051fab495baf98b1685bf5cb3cc7f1c8cbdfadf40fe320f9ae5ca367d"}
Sep 16 18:48:13.086587 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:13.086538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0db0dce-59ad-48d4-97fa-d97a54af1bce","Type":"ContainerStarted","Data":"33d050ffea400b226bbccea59bd086fa624aa19aadfc8bd54a2f2523daa93891"}
Sep 16 18:48:13.086587 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:13.086563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a0db0dce-59ad-48d4-97fa-d97a54af1bce","Type":"ContainerStarted","Data":"ebe32c9555d4ee6df71ff8416a31039a23037390f4ec66b3de9ad903b3f81a6d"}
Sep 16 18:48:13.135901 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:13.135850    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.116940078 podStartE2EDuration="2.135830156s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.890078ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:13.134744586 +0000 UTC m=+200.581839116" watchObservedRunningTime="2026-09-16 18:48:13.135830156 +0000 UTC m=+200.582924685"
Sep 16 18:48:16.461879 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:16.461834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:35.756330 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:48:35.756295    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69467445df-hwcxv"]
Sep 16 18:49:00.776381 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:00.776338    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69467445df-hwcxv" podUID="19009e13-f275-4830-b95d-4f4a98fdfec2" containerName="console" containerID="cri-o://ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7" gracePeriod=15
Sep 16 18:49:01.011341 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.011312    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:49:01.129157 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.129098    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp2xb\" (UniqueName: \"kubernetes.io/projected/19009e13-f275-4830-b95d-4f4a98fdfec2-kube-api-access-sp2xb\") pod \"19009e13-f275-4830-b95d-4f4a98fdfec2\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") "
Sep 16 18:49:01.129367 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.129173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-console-config\") pod \"19009e13-f275-4830-b95d-4f4a98fdfec2\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") "
Sep 16 18:49:01.129367 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.129195    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-trusted-ca-bundle\") pod \"19009e13-f275-4830-b95d-4f4a98fdfec2\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") "
Sep 16 18:49:01.129367 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.129246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-serving-cert\") pod \"19009e13-f275-4830-b95d-4f4a98fdfec2\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") "
Sep 16 18:49:01.129367 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.129268    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-oauth-config\") pod \"19009e13-f275-4830-b95d-4f4a98fdfec2\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") "
Sep 16 18:49:01.129367 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.129286    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-service-ca\") pod \"19009e13-f275-4830-b95d-4f4a98fdfec2\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") "
Sep 16 18:49:01.129367 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.129329    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-oauth-serving-cert\") pod \"19009e13-f275-4830-b95d-4f4a98fdfec2\" (UID: \"19009e13-f275-4830-b95d-4f4a98fdfec2\") "
Sep 16 18:49:01.129796 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.129767    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-console-config" (OuterVolumeSpecName: "console-config") pod "19009e13-f275-4830-b95d-4f4a98fdfec2" (UID: "19009e13-f275-4830-b95d-4f4a98fdfec2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:01.129974 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.129943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "19009e13-f275-4830-b95d-4f4a98fdfec2" (UID: "19009e13-f275-4830-b95d-4f4a98fdfec2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:01.130189 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.130003    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-service-ca" (OuterVolumeSpecName: "service-ca") pod "19009e13-f275-4830-b95d-4f4a98fdfec2" (UID: "19009e13-f275-4830-b95d-4f4a98fdfec2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:01.130297 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.130189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "19009e13-f275-4830-b95d-4f4a98fdfec2" (UID: "19009e13-f275-4830-b95d-4f4a98fdfec2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:01.131419 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.131384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19009e13-f275-4830-b95d-4f4a98fdfec2-kube-api-access-sp2xb" (OuterVolumeSpecName: "kube-api-access-sp2xb") pod "19009e13-f275-4830-b95d-4f4a98fdfec2" (UID: "19009e13-f275-4830-b95d-4f4a98fdfec2"). InnerVolumeSpecName "kube-api-access-sp2xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:01.131799 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.131765    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "19009e13-f275-4830-b95d-4f4a98fdfec2" (UID: "19009e13-f275-4830-b95d-4f4a98fdfec2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:01.131889 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.131838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "19009e13-f275-4830-b95d-4f4a98fdfec2" (UID: "19009e13-f275-4830-b95d-4f4a98fdfec2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:01.231050 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.231010    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-service-ca\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:49:01.231050 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.231041    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-oauth-serving-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:49:01.231050 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.231054    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp2xb\" (UniqueName: \"kubernetes.io/projected/19009e13-f275-4830-b95d-4f4a98fdfec2-kube-api-access-sp2xb\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:49:01.231320 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.231066    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-console-config\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:49:01.231320 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.231078    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19009e13-f275-4830-b95d-4f4a98fdfec2-trusted-ca-bundle\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:49:01.231320 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.231089    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-serving-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:49:01.231320 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.231100    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19009e13-f275-4830-b95d-4f4a98fdfec2-console-oauth-config\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:49:01.282021 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.281975    2578 generic.go:356] "Generic (PLEG): container finished" podID="19009e13-f275-4830-b95d-4f4a98fdfec2" containerID="ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7" exitCode=0
Sep 16 18:49:01.282216 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.282023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69467445df-hwcxv" event={"ID":"19009e13-f275-4830-b95d-4f4a98fdfec2","Type":"ContainerDied","Data":"ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7"}
Sep 16 18:49:01.282216 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.282041    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69467445df-hwcxv"
Sep 16 18:49:01.282216 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.282144    2578 scope.go:117] "RemoveContainer" containerID="ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7"
Sep 16 18:49:01.282370 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.282129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69467445df-hwcxv" event={"ID":"19009e13-f275-4830-b95d-4f4a98fdfec2","Type":"ContainerDied","Data":"6cfe9845f57cc2b386a26de73b15fb84b2148162fff4c3317508222df94f6414"}
Sep 16 18:49:01.291336 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.291316    2578 scope.go:117] "RemoveContainer" containerID="ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7"
Sep 16 18:49:01.291680 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:49:01.291662    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7\": container with ID starting with ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7 not found: ID does not exist" containerID="ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7"
Sep 16 18:49:01.291739 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.291688    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7"} err="failed to get container status \"ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7\": rpc error: code = NotFound desc = could not find container \"ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7\": container with ID starting with ac3ec8d358e08fecf718016d2b8f1430253d8336d822c81a833a2ba744d014a7 not found: ID does not exist"
Sep 16 18:49:01.304832 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.304803    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69467445df-hwcxv"]
Sep 16 18:49:01.309655 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:01.309628    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69467445df-hwcxv"]
Sep 16 18:49:03.063794 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:03.063760    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19009e13-f275-4830-b95d-4f4a98fdfec2" path="/var/lib/kubelet/pods/19009e13-f275-4830-b95d-4f4a98fdfec2/volumes"
Sep 16 18:49:11.461964 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:11.461926    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:11.477269 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:11.477246    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:12.342466 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:12.342440    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:52.946192 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:52.946166    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 18:49:52.949652 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:52.949625    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 18:49:52.950375 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:52.950356    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:49:52.954023 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:52.954005    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:49:52.958472 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:49:52.958455    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:51:03.858739 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.858653    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"]
Sep 16 18:51:03.859182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.859013    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19009e13-f275-4830-b95d-4f4a98fdfec2" containerName="console"
Sep 16 18:51:03.859182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.859024    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19009e13-f275-4830-b95d-4f4a98fdfec2" containerName="console"
Sep 16 18:51:03.859182 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.859086    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="19009e13-f275-4830-b95d-4f4a98fdfec2" containerName="console"
Sep 16 18:51:03.861138 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.861101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"
Sep 16 18:51:03.864289 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.864268    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pmsjq\""
Sep 16 18:51:03.864814 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.864797    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:51:03.865518 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.865469    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:51:03.885793 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.885760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"]
Sep 16 18:51:03.932171 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.932129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ae7b647-fbdc-44e9-a600-f96299ff63dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jtcs6\" (UID: \"6ae7b647-fbdc-44e9-a600-f96299ff63dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"
Sep 16 18:51:03.932357 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:03.932182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mkr\" (UniqueName: \"kubernetes.io/projected/6ae7b647-fbdc-44e9-a600-f96299ff63dc-kube-api-access-b5mkr\") pod \"cert-manager-cainjector-559cdd8c87-jtcs6\" (UID: \"6ae7b647-fbdc-44e9-a600-f96299ff63dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"
Sep 16 18:51:04.033508 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:04.033434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ae7b647-fbdc-44e9-a600-f96299ff63dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jtcs6\" (UID: \"6ae7b647-fbdc-44e9-a600-f96299ff63dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"
Sep 16 18:51:04.033692 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:04.033543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mkr\" (UniqueName: \"kubernetes.io/projected/6ae7b647-fbdc-44e9-a600-f96299ff63dc-kube-api-access-b5mkr\") pod \"cert-manager-cainjector-559cdd8c87-jtcs6\" (UID: \"6ae7b647-fbdc-44e9-a600-f96299ff63dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"
Sep 16 18:51:04.044009 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:04.043980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mkr\" (UniqueName: \"kubernetes.io/projected/6ae7b647-fbdc-44e9-a600-f96299ff63dc-kube-api-access-b5mkr\") pod \"cert-manager-cainjector-559cdd8c87-jtcs6\" (UID: \"6ae7b647-fbdc-44e9-a600-f96299ff63dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"
Sep 16 18:51:04.044875 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:04.044853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ae7b647-fbdc-44e9-a600-f96299ff63dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jtcs6\" (UID: \"6ae7b647-fbdc-44e9-a600-f96299ff63dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"
Sep 16 18:51:04.181541 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:04.181418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"
Sep 16 18:51:04.465280 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:04.465253    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6"]
Sep 16 18:51:04.466846 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:51:04.466816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ae7b647_fbdc_44e9_a600_f96299ff63dc.slice/crio-d173e0771b9ab6ca0311ab546cd505bd98c70019f345714c2cee69064548167a WatchSource:0}: Error finding container d173e0771b9ab6ca0311ab546cd505bd98c70019f345714c2cee69064548167a: Status 404 returned error can't find the container with id d173e0771b9ab6ca0311ab546cd505bd98c70019f345714c2cee69064548167a
Sep 16 18:51:04.468804 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:04.468783    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:51:04.775264 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:04.775182    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6" event={"ID":"6ae7b647-fbdc-44e9-a600-f96299ff63dc","Type":"ContainerStarted","Data":"d173e0771b9ab6ca0311ab546cd505bd98c70019f345714c2cee69064548167a"}
Sep 16 18:51:07.790338 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:07.790297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6" event={"ID":"6ae7b647-fbdc-44e9-a600-f96299ff63dc","Type":"ContainerStarted","Data":"47e4b00979fcc68c85721bea4ca008fa07e229566d546364f15b87365e59d676"}
Sep 16 18:51:07.812509 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:07.812439    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jtcs6" podStartSLOduration=1.564225566 podStartE2EDuration="4.812423609s" totalImagesPullingTime="3.248198043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:07.811558028 +0000 UTC m=+375.258652579" watchObservedRunningTime="2026-09-16 18:51:07.812423609 +0000 UTC m=+375.259518138"
Sep 16 18:51:12.373077 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.373030    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"]
Sep 16 18:51:12.376006 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.375987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"
Sep 16 18:51:12.378886 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.378866    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:51:12.380014 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.379991    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-4ntcd\""
Sep 16 18:51:12.380128 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.380025    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:51:12.385921 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.385896    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"]
Sep 16 18:51:12.508118 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.508078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcgx\" (UniqueName: \"kubernetes.io/projected/892f724b-9e97-4bf9-8072-2d3e690bbfd5-kube-api-access-lkcgx\") pod \"openshift-lws-operator-fd8ccff4c-ksxq9\" (UID: \"892f724b-9e97-4bf9-8072-2d3e690bbfd5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"
Sep 16 18:51:12.508285 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.508141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/892f724b-9e97-4bf9-8072-2d3e690bbfd5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ksxq9\" (UID: \"892f724b-9e97-4bf9-8072-2d3e690bbfd5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"
Sep 16 18:51:12.609076 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.609041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/892f724b-9e97-4bf9-8072-2d3e690bbfd5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ksxq9\" (UID: \"892f724b-9e97-4bf9-8072-2d3e690bbfd5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"
Sep 16 18:51:12.609261 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.609113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcgx\" (UniqueName: \"kubernetes.io/projected/892f724b-9e97-4bf9-8072-2d3e690bbfd5-kube-api-access-lkcgx\") pod \"openshift-lws-operator-fd8ccff4c-ksxq9\" (UID: \"892f724b-9e97-4bf9-8072-2d3e690bbfd5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"
Sep 16 18:51:12.609438 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.609417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/892f724b-9e97-4bf9-8072-2d3e690bbfd5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ksxq9\" (UID: \"892f724b-9e97-4bf9-8072-2d3e690bbfd5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"
Sep 16 18:51:12.618160 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.618135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcgx\" (UniqueName: \"kubernetes.io/projected/892f724b-9e97-4bf9-8072-2d3e690bbfd5-kube-api-access-lkcgx\") pod \"openshift-lws-operator-fd8ccff4c-ksxq9\" (UID: \"892f724b-9e97-4bf9-8072-2d3e690bbfd5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"
Sep 16 18:51:12.685351 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.685252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"
Sep 16 18:51:12.956267 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:12.956240    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9"]
Sep 16 18:51:12.959005 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:51:12.958977    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod892f724b_9e97_4bf9_8072_2d3e690bbfd5.slice/crio-262e503a559dc0ee9669c3e10d38c95e2415dbd10b5dc58f44950bf69da3992a WatchSource:0}: Error finding container 262e503a559dc0ee9669c3e10d38c95e2415dbd10b5dc58f44950bf69da3992a: Status 404 returned error can't find the container with id 262e503a559dc0ee9669c3e10d38c95e2415dbd10b5dc58f44950bf69da3992a
Sep 16 18:51:13.818836 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:13.818777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9" event={"ID":"892f724b-9e97-4bf9-8072-2d3e690bbfd5","Type":"ContainerStarted","Data":"262e503a559dc0ee9669c3e10d38c95e2415dbd10b5dc58f44950bf69da3992a"}
Sep 16 18:51:15.830068 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:15.830029    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9" event={"ID":"892f724b-9e97-4bf9-8072-2d3e690bbfd5","Type":"ContainerStarted","Data":"06eaa18625a35747f9eea3ec667a32a01b085d1ddac8d1755f9e39d4dd12443e"}
Sep 16 18:51:15.848642 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:15.848589    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ksxq9" podStartSLOduration=1.562630799 podStartE2EDuration="3.848573569s" totalImagesPullingTime="2.28594277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:15.848455262 +0000 UTC m=+383.295549793" watchObservedRunningTime="2026-09-16 18:51:15.848573569 +0000 UTC m=+383.295668099"
Sep 16 18:51:37.852905 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.852859    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"]
Sep 16 18:51:37.856236 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.856219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:37.861282 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.861255    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:51:37.863157 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.863132    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:51:37.864364 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.864343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:51:37.864609 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.864593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mkdck\""
Sep 16 18:51:37.876561 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.876531    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"]
Sep 16 18:51:37.898222 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.898192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b9d9397-e72b-4637-9384-d5df58fc4c3a-manager-config\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:37.898366 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.898231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x2fp\" (UniqueName: \"kubernetes.io/projected/9b9d9397-e72b-4637-9384-d5df58fc4c3a-kube-api-access-8x2fp\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:37.898366 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.898308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b9d9397-e72b-4637-9384-d5df58fc4c3a-cert\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:37.898366 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.898348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b9d9397-e72b-4637-9384-d5df58fc4c3a-metrics-cert\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:37.999252 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.999210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b9d9397-e72b-4637-9384-d5df58fc4c3a-metrics-cert\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:37.999252 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.999250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b9d9397-e72b-4637-9384-d5df58fc4c3a-manager-config\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:37.999613 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.999276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x2fp\" (UniqueName: \"kubernetes.io/projected/9b9d9397-e72b-4637-9384-d5df58fc4c3a-kube-api-access-8x2fp\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:37.999613 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.999328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b9d9397-e72b-4637-9384-d5df58fc4c3a-cert\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:38.000012 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:37.999990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b9d9397-e72b-4637-9384-d5df58fc4c3a-manager-config\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:38.001916 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:38.001889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b9d9397-e72b-4637-9384-d5df58fc4c3a-cert\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:38.002018 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:38.001956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b9d9397-e72b-4637-9384-d5df58fc4c3a-metrics-cert\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:38.026660 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:38.026631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x2fp\" (UniqueName: \"kubernetes.io/projected/9b9d9397-e72b-4637-9384-d5df58fc4c3a-kube-api-access-8x2fp\") pod \"lws-controller-manager-796cdc484c-94n74\" (UID: \"9b9d9397-e72b-4637-9384-d5df58fc4c3a\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:38.166067 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:38.165962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:38.459138 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:38.459110    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"]
Sep 16 18:51:38.461233 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:51:38.461201    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9d9397_e72b_4637_9384_d5df58fc4c3a.slice/crio-f0bec6ab33bb2ad90940aaf3b837de91f95a9a4f7a16da575b0a34ea7b55463e WatchSource:0}: Error finding container f0bec6ab33bb2ad90940aaf3b837de91f95a9a4f7a16da575b0a34ea7b55463e: Status 404 returned error can't find the container with id f0bec6ab33bb2ad90940aaf3b837de91f95a9a4f7a16da575b0a34ea7b55463e
Sep 16 18:51:38.924973 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:38.924917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74" event={"ID":"9b9d9397-e72b-4637-9384-d5df58fc4c3a","Type":"ContainerStarted","Data":"f0bec6ab33bb2ad90940aaf3b837de91f95a9a4f7a16da575b0a34ea7b55463e"}
Sep 16 18:51:39.930591 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:39.930557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74" event={"ID":"9b9d9397-e72b-4637-9384-d5df58fc4c3a","Type":"ContainerStarted","Data":"fe01d2f96b18f45800f56ee98944414f85c1ecba91a7307aa1c97f7260225aa1"}
Sep 16 18:51:39.930975 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:39.930622    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:51:39.957658 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:39.957600    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74" podStartSLOduration=1.694612259 podStartE2EDuration="2.957581587s" totalImagesPullingTime="1.262969328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:39.954343462 +0000 UTC m=+407.401437991" watchObservedRunningTime="2026-09-16 18:51:39.957581587 +0000 UTC m=+407.404676116"
Sep 16 18:51:50.937378 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:51:50.937342    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-94n74"
Sep 16 18:52:07.279918 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.279882    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"]
Sep 16 18:52:07.289550 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.289514    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"
Sep 16 18:52:07.295204 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.295007    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:52:07.295204 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.295023    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:52:07.295204 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.295023    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rh822\""
Sep 16 18:52:07.298003 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.297981    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"]
Sep 16 18:52:07.337850 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.337820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8mjg\" (UniqueName: \"kubernetes.io/projected/736deb6d-692f-4e80-b283-cfd74634c900-kube-api-access-l8mjg\") pod \"authorino-operator-6fc8f864f4-wrrqb\" (UID: \"736deb6d-692f-4e80-b283-cfd74634c900\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"
Sep 16 18:52:07.439164 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.439125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8mjg\" (UniqueName: \"kubernetes.io/projected/736deb6d-692f-4e80-b283-cfd74634c900-kube-api-access-l8mjg\") pod \"authorino-operator-6fc8f864f4-wrrqb\" (UID: \"736deb6d-692f-4e80-b283-cfd74634c900\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"
Sep 16 18:52:07.448102 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.448075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8mjg\" (UniqueName: \"kubernetes.io/projected/736deb6d-692f-4e80-b283-cfd74634c900-kube-api-access-l8mjg\") pod \"authorino-operator-6fc8f864f4-wrrqb\" (UID: \"736deb6d-692f-4e80-b283-cfd74634c900\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"
Sep 16 18:52:07.599820 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.599738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"
Sep 16 18:52:07.882737 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:07.882711    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"]
Sep 16 18:52:07.884704 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:52:07.884619    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736deb6d_692f_4e80_b283_cfd74634c900.slice/crio-ac4a542f5f420814ee20e4216c821712c8f904beefaacf3a41806664fd695849 WatchSource:0}: Error finding container ac4a542f5f420814ee20e4216c821712c8f904beefaacf3a41806664fd695849: Status 404 returned error can't find the container with id ac4a542f5f420814ee20e4216c821712c8f904beefaacf3a41806664fd695849
Sep 16 18:52:08.051709 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:08.051673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb" event={"ID":"736deb6d-692f-4e80-b283-cfd74634c900","Type":"ContainerStarted","Data":"ac4a542f5f420814ee20e4216c821712c8f904beefaacf3a41806664fd695849"}
Sep 16 18:52:11.068439 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:11.068402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb" event={"ID":"736deb6d-692f-4e80-b283-cfd74634c900","Type":"ContainerStarted","Data":"a9f5c0845aad338df51362b585f75cc829a9e4256d970a844ba9148e55c500c9"}
Sep 16 18:52:11.068995 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:11.068518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"
Sep 16 18:52:11.089208 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:11.089155    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb" podStartSLOduration=1.937641041 podStartE2EDuration="4.089139438s" totalImagesPullingTime="2.151498397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:11.088334442 +0000 UTC m=+438.535428972" watchObservedRunningTime="2026-09-16 18:52:11.089139438 +0000 UTC m=+438.536233967"
Sep 16 18:52:22.075929 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:22.075848    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-wrrqb"
Sep 16 18:52:56.963916 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:56.963885    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hf4mb"]
Sep 16 18:52:56.967530 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:56.967507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb"
Sep 16 18:52:56.970618 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:56.970596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pkdmw\""
Sep 16 18:52:56.977049 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:56.977028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hf4mb"]
Sep 16 18:52:57.056376 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:57.056341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb6x\" (UniqueName: \"kubernetes.io/projected/59dd5017-6626-4b61-b6f0-25f0776365b9-kube-api-access-8mb6x\") pod \"authorino-7bcfd8f9cd-hf4mb\" (UID: \"59dd5017-6626-4b61-b6f0-25f0776365b9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb"
Sep 16 18:52:57.157238 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:57.157204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb6x\" (UniqueName: \"kubernetes.io/projected/59dd5017-6626-4b61-b6f0-25f0776365b9-kube-api-access-8mb6x\") pod \"authorino-7bcfd8f9cd-hf4mb\" (UID: \"59dd5017-6626-4b61-b6f0-25f0776365b9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb"
Sep 16 18:52:57.165233 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:57.165199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb6x\" (UniqueName: \"kubernetes.io/projected/59dd5017-6626-4b61-b6f0-25f0776365b9-kube-api-access-8mb6x\") pod \"authorino-7bcfd8f9cd-hf4mb\" (UID: \"59dd5017-6626-4b61-b6f0-25f0776365b9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb"
Sep 16 18:52:57.277512 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:57.277411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb"
Sep 16 18:52:57.545964 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:57.545938    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hf4mb"]
Sep 16 18:52:57.548179 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:52:57.548153    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59dd5017_6626_4b61_b6f0_25f0776365b9.slice/crio-c721c2865a2077171fe8a91e051c5663c5c3f8b8acca934480efb6d3f896293a WatchSource:0}: Error finding container c721c2865a2077171fe8a91e051c5663c5c3f8b8acca934480efb6d3f896293a: Status 404 returned error can't find the container with id c721c2865a2077171fe8a91e051c5663c5c3f8b8acca934480efb6d3f896293a
Sep 16 18:52:58.277278 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:52:58.277243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb" event={"ID":"59dd5017-6626-4b61-b6f0-25f0776365b9","Type":"ContainerStarted","Data":"c721c2865a2077171fe8a91e051c5663c5c3f8b8acca934480efb6d3f896293a"}
Sep 16 18:53:00.123522 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:00.123474    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hf4mb"]
Sep 16 18:53:00.287042 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:00.287009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb" event={"ID":"59dd5017-6626-4b61-b6f0-25f0776365b9","Type":"ContainerStarted","Data":"05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c"}
Sep 16 18:53:00.304039 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:00.303991    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb" podStartSLOduration=2.604678342 podStartE2EDuration="4.303975863s" totalImagesPullingTime="1.699297521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:00.302816309 +0000 UTC m=+487.749910838" watchObservedRunningTime="2026-09-16 18:53:00.303975863 +0000 UTC m=+487.751070390"
Sep 16 18:53:01.292234 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:01.292170    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb" podUID="59dd5017-6626-4b61-b6f0-25f0776365b9" containerName="authorino" containerID="cri-o://05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c" gracePeriod=30
Sep 16 18:53:01.536386 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:01.536355    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb"
Sep 16 18:53:01.599299 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:01.599210    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mb6x\" (UniqueName: \"kubernetes.io/projected/59dd5017-6626-4b61-b6f0-25f0776365b9-kube-api-access-8mb6x\") pod \"59dd5017-6626-4b61-b6f0-25f0776365b9\" (UID: \"59dd5017-6626-4b61-b6f0-25f0776365b9\") "
Sep 16 18:53:01.601407 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:01.601379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dd5017-6626-4b61-b6f0-25f0776365b9-kube-api-access-8mb6x" (OuterVolumeSpecName: "kube-api-access-8mb6x") pod "59dd5017-6626-4b61-b6f0-25f0776365b9" (UID: "59dd5017-6626-4b61-b6f0-25f0776365b9"). InnerVolumeSpecName "kube-api-access-8mb6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:01.699980 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:01.699942    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mb6x\" (UniqueName: \"kubernetes.io/projected/59dd5017-6626-4b61-b6f0-25f0776365b9-kube-api-access-8mb6x\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:53:02.298026 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:02.297994    2578 generic.go:356] "Generic (PLEG): container finished" podID="59dd5017-6626-4b61-b6f0-25f0776365b9" containerID="05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c" exitCode=0
Sep 16 18:53:02.298026 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:02.298033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb" event={"ID":"59dd5017-6626-4b61-b6f0-25f0776365b9","Type":"ContainerDied","Data":"05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c"}
Sep 16 18:53:02.298618 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:02.298039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb"
Sep 16 18:53:02.298618 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:02.298061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hf4mb" event={"ID":"59dd5017-6626-4b61-b6f0-25f0776365b9","Type":"ContainerDied","Data":"c721c2865a2077171fe8a91e051c5663c5c3f8b8acca934480efb6d3f896293a"}
Sep 16 18:53:02.298618 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:02.298081    2578 scope.go:117] "RemoveContainer" containerID="05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c"
Sep 16 18:53:02.309740 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:02.309717    2578 scope.go:117] "RemoveContainer" containerID="05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c"
Sep 16 18:53:02.310093 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:53:02.310068    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c\": container with ID starting with 05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c not found: ID does not exist" containerID="05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c"
Sep 16 18:53:02.310159 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:02.310106    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c"} err="failed to get container status \"05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c\": rpc error: code = NotFound desc = could not find container \"05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c\": container with ID starting with 05c2c1ce57f1eb5e6ba8e8fd05d5bd8cbfc61a16f2d19c79a4af6d497e73f29c not found: ID does not exist"
Sep 16 18:53:02.319027 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:02.319000    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hf4mb"]
Sep 16 18:53:02.321312 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:02.321289    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hf4mb"]
Sep 16 18:53:03.064609 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:03.064571    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59dd5017-6626-4b61-b6f0-25f0776365b9" path="/var/lib/kubelet/pods/59dd5017-6626-4b61-b6f0-25f0776365b9/volumes"
Sep 16 18:53:55.693033 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.693000    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-8x8b5"]
Sep 16 18:53:55.693601 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.693370    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59dd5017-6626-4b61-b6f0-25f0776365b9" containerName="authorino"
Sep 16 18:53:55.693601 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.693381    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="59dd5017-6626-4b61-b6f0-25f0776365b9" containerName="authorino"
Sep 16 18:53:55.693601 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.693436    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="59dd5017-6626-4b61-b6f0-25f0776365b9" containerName="authorino"
Sep 16 18:53:55.696685 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.696659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:55.699409 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.699385    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:53:55.699551 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.699429    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:53:55.700267 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.700250    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 18:53:55.700351 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.700282    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-4hrn8\""
Sep 16 18:53:55.708924 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.708897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-8x8b5"]
Sep 16 18:53:55.765501 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.765443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7854953a-f045-4510-b61e-5fc870cadd2e-cert\") pod \"kserve-controller-manager-75c4c6c467-8x8b5\" (UID: \"7854953a-f045-4510-b61e-5fc870cadd2e\") " pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:55.765671 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.765567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tkzw\" (UniqueName: \"kubernetes.io/projected/7854953a-f045-4510-b61e-5fc870cadd2e-kube-api-access-8tkzw\") pod \"kserve-controller-manager-75c4c6c467-8x8b5\" (UID: \"7854953a-f045-4510-b61e-5fc870cadd2e\") " pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:55.866550 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.866508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7854953a-f045-4510-b61e-5fc870cadd2e-cert\") pod \"kserve-controller-manager-75c4c6c467-8x8b5\" (UID: \"7854953a-f045-4510-b61e-5fc870cadd2e\") " pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:55.866737 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.866615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tkzw\" (UniqueName: \"kubernetes.io/projected/7854953a-f045-4510-b61e-5fc870cadd2e-kube-api-access-8tkzw\") pod \"kserve-controller-manager-75c4c6c467-8x8b5\" (UID: \"7854953a-f045-4510-b61e-5fc870cadd2e\") " pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:55.866737 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:53:55.866686    2578 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 18:53:55.866861 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:53:55.866769    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7854953a-f045-4510-b61e-5fc870cadd2e-cert podName:7854953a-f045-4510-b61e-5fc870cadd2e nodeName:}" failed. No retries permitted until 2026-09-16 18:53:56.366747563 +0000 UTC m=+543.813842076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7854953a-f045-4510-b61e-5fc870cadd2e-cert") pod "kserve-controller-manager-75c4c6c467-8x8b5" (UID: "7854953a-f045-4510-b61e-5fc870cadd2e") : secret "kserve-webhook-server-cert" not found
Sep 16 18:53:55.885738 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:55.885710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tkzw\" (UniqueName: \"kubernetes.io/projected/7854953a-f045-4510-b61e-5fc870cadd2e-kube-api-access-8tkzw\") pod \"kserve-controller-manager-75c4c6c467-8x8b5\" (UID: \"7854953a-f045-4510-b61e-5fc870cadd2e\") " pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:56.370797 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:56.370753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7854953a-f045-4510-b61e-5fc870cadd2e-cert\") pod \"kserve-controller-manager-75c4c6c467-8x8b5\" (UID: \"7854953a-f045-4510-b61e-5fc870cadd2e\") " pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:56.373277 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:56.373242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7854953a-f045-4510-b61e-5fc870cadd2e-cert\") pod \"kserve-controller-manager-75c4c6c467-8x8b5\" (UID: \"7854953a-f045-4510-b61e-5fc870cadd2e\") " pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:56.609279 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:56.609235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:56.879726 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:56.879698    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-8x8b5"]
Sep 16 18:53:56.881599 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:53:56.881570    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7854953a_f045_4510_b61e_5fc870cadd2e.slice/crio-9c682e9ba81002a14301040c142956f6995e81a31d703cbd9cdfb932a34d0fda WatchSource:0}: Error finding container 9c682e9ba81002a14301040c142956f6995e81a31d703cbd9cdfb932a34d0fda: Status 404 returned error can't find the container with id 9c682e9ba81002a14301040c142956f6995e81a31d703cbd9cdfb932a34d0fda
Sep 16 18:53:57.552452 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:57.552416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5" event={"ID":"7854953a-f045-4510-b61e-5fc870cadd2e","Type":"ContainerStarted","Data":"9c682e9ba81002a14301040c142956f6995e81a31d703cbd9cdfb932a34d0fda"}
Sep 16 18:53:59.563340 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:59.563301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5" event={"ID":"7854953a-f045-4510-b61e-5fc870cadd2e","Type":"ContainerStarted","Data":"c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a"}
Sep 16 18:53:59.563809 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:59.563398    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:53:59.581568 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:53:59.581521    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5" podStartSLOduration=2.1274837460000002 podStartE2EDuration="4.581505508s" totalImagesPullingTime="2.454021762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:59.58043562 +0000 UTC m=+547.027530163" watchObservedRunningTime="2026-09-16 18:53:59.581505508 +0000 UTC m=+547.028600037"
Sep 16 18:54:30.573757 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:30.573725    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:54:40.697309 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.697271    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-8x8b5"]
Sep 16 18:54:40.697824 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.697596    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5" podUID="7854953a-f045-4510-b61e-5fc870cadd2e" containerName="manager" containerID="cri-o://c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a" gracePeriod=10
Sep 16 18:54:40.725090 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.725063    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-hb627"]
Sep 16 18:54:40.727353 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.727335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:54:40.736771 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.736747    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-hb627"]
Sep 16 18:54:40.864962 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.864921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fps46\" (UniqueName: \"kubernetes.io/projected/185e38f2-6449-4cda-ab93-632d097a2278-kube-api-access-fps46\") pod \"kserve-controller-manager-75c4c6c467-hb627\" (UID: \"185e38f2-6449-4cda-ab93-632d097a2278\") " pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:54:40.865127 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.865093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/185e38f2-6449-4cda-ab93-632d097a2278-cert\") pod \"kserve-controller-manager-75c4c6c467-hb627\" (UID: \"185e38f2-6449-4cda-ab93-632d097a2278\") " pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:54:40.948137 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.948079    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:54:40.965899 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.965868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/185e38f2-6449-4cda-ab93-632d097a2278-cert\") pod \"kserve-controller-manager-75c4c6c467-hb627\" (UID: \"185e38f2-6449-4cda-ab93-632d097a2278\") " pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:54:40.966075 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.965944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fps46\" (UniqueName: \"kubernetes.io/projected/185e38f2-6449-4cda-ab93-632d097a2278-kube-api-access-fps46\") pod \"kserve-controller-manager-75c4c6c467-hb627\" (UID: \"185e38f2-6449-4cda-ab93-632d097a2278\") " pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:54:40.968628 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.968598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/185e38f2-6449-4cda-ab93-632d097a2278-cert\") pod \"kserve-controller-manager-75c4c6c467-hb627\" (UID: \"185e38f2-6449-4cda-ab93-632d097a2278\") " pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:54:40.977761 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:40.977732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fps46\" (UniqueName: \"kubernetes.io/projected/185e38f2-6449-4cda-ab93-632d097a2278-kube-api-access-fps46\") pod \"kserve-controller-manager-75c4c6c467-hb627\" (UID: \"185e38f2-6449-4cda-ab93-632d097a2278\") " pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:54:41.066578 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.066547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tkzw\" (UniqueName: \"kubernetes.io/projected/7854953a-f045-4510-b61e-5fc870cadd2e-kube-api-access-8tkzw\") pod \"7854953a-f045-4510-b61e-5fc870cadd2e\" (UID: \"7854953a-f045-4510-b61e-5fc870cadd2e\") "
Sep 16 18:54:41.066728 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.066592    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7854953a-f045-4510-b61e-5fc870cadd2e-cert\") pod \"7854953a-f045-4510-b61e-5fc870cadd2e\" (UID: \"7854953a-f045-4510-b61e-5fc870cadd2e\") "
Sep 16 18:54:41.068087 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.068063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:54:41.068762 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.068733    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7854953a-f045-4510-b61e-5fc870cadd2e-cert" (OuterVolumeSpecName: "cert") pod "7854953a-f045-4510-b61e-5fc870cadd2e" (UID: "7854953a-f045-4510-b61e-5fc870cadd2e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:41.068845 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.068754    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7854953a-f045-4510-b61e-5fc870cadd2e-kube-api-access-8tkzw" (OuterVolumeSpecName: "kube-api-access-8tkzw") pod "7854953a-f045-4510-b61e-5fc870cadd2e" (UID: "7854953a-f045-4510-b61e-5fc870cadd2e"). InnerVolumeSpecName "kube-api-access-8tkzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:41.167438 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.167392    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tkzw\" (UniqueName: \"kubernetes.io/projected/7854953a-f045-4510-b61e-5fc870cadd2e-kube-api-access-8tkzw\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:54:41.167438 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.167432    2578 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7854953a-f045-4510-b61e-5fc870cadd2e-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:54:41.335720 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.335696    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-hb627"]
Sep 16 18:54:41.337934 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:54:41.337898    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185e38f2_6449_4cda_ab93_632d097a2278.slice/crio-c38e68fe7be1a1b9bb258306ba8cf95e9adf8cf3604fd1d4c0dde04b2bbe7726 WatchSource:0}: Error finding container c38e68fe7be1a1b9bb258306ba8cf95e9adf8cf3604fd1d4c0dde04b2bbe7726: Status 404 returned error can't find the container with id c38e68fe7be1a1b9bb258306ba8cf95e9adf8cf3604fd1d4c0dde04b2bbe7726
Sep 16 18:54:41.750336 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.750199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-hb627" event={"ID":"185e38f2-6449-4cda-ab93-632d097a2278","Type":"ContainerStarted","Data":"c38e68fe7be1a1b9bb258306ba8cf95e9adf8cf3604fd1d4c0dde04b2bbe7726"}
Sep 16 18:54:41.751510 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.751462    2578 generic.go:356] "Generic (PLEG): container finished" podID="7854953a-f045-4510-b61e-5fc870cadd2e" containerID="c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a" exitCode=0
Sep 16 18:54:41.751594 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.751537    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5"
Sep 16 18:54:41.751647 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.751590    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5" event={"ID":"7854953a-f045-4510-b61e-5fc870cadd2e","Type":"ContainerDied","Data":"c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a"}
Sep 16 18:54:41.751647 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.751625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-8x8b5" event={"ID":"7854953a-f045-4510-b61e-5fc870cadd2e","Type":"ContainerDied","Data":"9c682e9ba81002a14301040c142956f6995e81a31d703cbd9cdfb932a34d0fda"}
Sep 16 18:54:41.751647 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.751644    2578 scope.go:117] "RemoveContainer" containerID="c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a"
Sep 16 18:54:41.760940 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.760915    2578 scope.go:117] "RemoveContainer" containerID="c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a"
Sep 16 18:54:41.761285 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:54:41.761256    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a\": container with ID starting with c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a not found: ID does not exist" containerID="c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a"
Sep 16 18:54:41.761398 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.761295    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a"} err="failed to get container status \"c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a\": rpc error: code = NotFound desc = could not find container \"c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a\": container with ID starting with c829417d7473f0c01c06449bec5a24fc8e68fe2992e1c34e77772cf2e1d0991a not found: ID does not exist"
Sep 16 18:54:41.775448 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.775400    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-8x8b5"]
Sep 16 18:54:41.780055 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:41.780020    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-8x8b5"]
Sep 16 18:54:42.758779 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:42.758744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-hb627" event={"ID":"185e38f2-6449-4cda-ab93-632d097a2278","Type":"ContainerStarted","Data":"bde53786d7d08c970146db5ea947e1992aa03c042278b51788ce24a578655373"}
Sep 16 18:54:42.759153 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:42.758869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:54:42.776471 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:42.776423    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-75c4c6c467-hb627" podStartSLOduration=2.4184185 podStartE2EDuration="2.776408927s" totalImagesPullingTime="357.990427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:42.775516726 +0000 UTC m=+590.222611255" watchObservedRunningTime="2026-09-16 18:54:42.776408927 +0000 UTC m=+590.223503456"
Sep 16 18:54:43.063798 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:43.063757    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7854953a-f045-4510-b61e-5fc870cadd2e" path="/var/lib/kubelet/pods/7854953a-f045-4510-b61e-5fc870cadd2e/volumes"
Sep 16 18:54:52.979258 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:52.979231    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 18:54:52.980028 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:52.980005    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 18:54:52.983012 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:52.982993    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:54:52.983649 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:54:52.983631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:55:13.767681 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:13.767649    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-75c4c6c467-hb627"
Sep 16 18:55:15.163126 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.163038    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-mp2wf"]
Sep 16 18:55:15.163623 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.163606    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7854953a-f045-4510-b61e-5fc870cadd2e" containerName="manager"
Sep 16 18:55:15.163675 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.163626    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7854953a-f045-4510-b61e-5fc870cadd2e" containerName="manager"
Sep 16 18:55:15.163746 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.163736    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7854953a-f045-4510-b61e-5fc870cadd2e" containerName="manager"
Sep 16 18:55:15.168319 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.168297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.171213 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.171189    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:55:15.171353 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.171248    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-ghdzr\""
Sep 16 18:55:15.172305 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.172285    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:55:15.176647 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.176440    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-mp2wf"]
Sep 16 18:55:15.259975 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.259941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5e1865e-3bfb-4e67-9cee-5d52d60812ce-cert\") pod \"odh-model-controller-7b966d7984-mp2wf\" (UID: \"d5e1865e-3bfb-4e67-9cee-5d52d60812ce\") " pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.260148 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.260003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnr7h\" (UniqueName: \"kubernetes.io/projected/d5e1865e-3bfb-4e67-9cee-5d52d60812ce-kube-api-access-lnr7h\") pod \"odh-model-controller-7b966d7984-mp2wf\" (UID: \"d5e1865e-3bfb-4e67-9cee-5d52d60812ce\") " pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.260148 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.260042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e1865e-3bfb-4e67-9cee-5d52d60812ce-metrics-certs\") pod \"odh-model-controller-7b966d7984-mp2wf\" (UID: \"d5e1865e-3bfb-4e67-9cee-5d52d60812ce\") " pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.361211 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.361175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5e1865e-3bfb-4e67-9cee-5d52d60812ce-cert\") pod \"odh-model-controller-7b966d7984-mp2wf\" (UID: \"d5e1865e-3bfb-4e67-9cee-5d52d60812ce\") " pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.361392 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.361232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnr7h\" (UniqueName: \"kubernetes.io/projected/d5e1865e-3bfb-4e67-9cee-5d52d60812ce-kube-api-access-lnr7h\") pod \"odh-model-controller-7b966d7984-mp2wf\" (UID: \"d5e1865e-3bfb-4e67-9cee-5d52d60812ce\") " pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.361392 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.361267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e1865e-3bfb-4e67-9cee-5d52d60812ce-metrics-certs\") pod \"odh-model-controller-7b966d7984-mp2wf\" (UID: \"d5e1865e-3bfb-4e67-9cee-5d52d60812ce\") " pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.363777 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.363750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5e1865e-3bfb-4e67-9cee-5d52d60812ce-cert\") pod \"odh-model-controller-7b966d7984-mp2wf\" (UID: \"d5e1865e-3bfb-4e67-9cee-5d52d60812ce\") " pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.363901 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.363783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e1865e-3bfb-4e67-9cee-5d52d60812ce-metrics-certs\") pod \"odh-model-controller-7b966d7984-mp2wf\" (UID: \"d5e1865e-3bfb-4e67-9cee-5d52d60812ce\") " pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.370253 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.370222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnr7h\" (UniqueName: \"kubernetes.io/projected/d5e1865e-3bfb-4e67-9cee-5d52d60812ce-kube-api-access-lnr7h\") pod \"odh-model-controller-7b966d7984-mp2wf\" (UID: \"d5e1865e-3bfb-4e67-9cee-5d52d60812ce\") " pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.481640 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.481552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:15.752600 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.752574    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-mp2wf"]
Sep 16 18:55:15.754423 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:55:15.754397    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e1865e_3bfb_4e67_9cee_5d52d60812ce.slice/crio-b6ecf081ef19ba08761a58541c173b595da7ef5959a1d3b9228a6dfb7582b604 WatchSource:0}: Error finding container b6ecf081ef19ba08761a58541c173b595da7ef5959a1d3b9228a6dfb7582b604: Status 404 returned error can't find the container with id b6ecf081ef19ba08761a58541c173b595da7ef5959a1d3b9228a6dfb7582b604
Sep 16 18:55:15.907155 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:15.907121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-mp2wf" event={"ID":"d5e1865e-3bfb-4e67-9cee-5d52d60812ce","Type":"ContainerStarted","Data":"b6ecf081ef19ba08761a58541c173b595da7ef5959a1d3b9228a6dfb7582b604"}
Sep 16 18:55:18.929941 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:18.929906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-mp2wf" event={"ID":"d5e1865e-3bfb-4e67-9cee-5d52d60812ce","Type":"ContainerStarted","Data":"6da46997efeec22dae37359bdf21fec63f73b773225c5ef5011274c1d8c2b6d0"}
Sep 16 18:55:18.930325 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:18.930039    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:18.947195 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:18.947116    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-mp2wf" podStartSLOduration=1.603702698 podStartE2EDuration="3.947099611s" totalImagesPullingTime="2.343396913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:18.946816248 +0000 UTC m=+626.393910778" watchObservedRunningTime="2026-09-16 18:55:18.947099611 +0000 UTC m=+626.394194187"
Sep 16 18:55:29.937432 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:29.937397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-mp2wf"
Sep 16 18:55:30.775501 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:30.775457    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-jx5ws"]
Sep 16 18:55:30.778989 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:30.778973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jx5ws"
Sep 16 18:55:30.782125 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:30.782107    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:55:30.782185 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:30.782137    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-k8879\""
Sep 16 18:55:30.782237 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:30.782107    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:55:30.787899 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:30.787876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jx5ws"]
Sep 16 18:55:30.901660 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:30.901621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2qb\" (UniqueName: \"kubernetes.io/projected/39d82901-ba40-4aed-a787-8d02b9a6db70-kube-api-access-7p2qb\") pod \"s3-init-jx5ws\" (UID: \"39d82901-ba40-4aed-a787-8d02b9a6db70\") " pod="kserve/s3-init-jx5ws"
Sep 16 18:55:30.901831 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:30.901671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d82901-ba40-4aed-a787-8d02b9a6db70-model-cache\") pod \"s3-init-jx5ws\" (UID: \"39d82901-ba40-4aed-a787-8d02b9a6db70\") " pod="kserve/s3-init-jx5ws"
Sep 16 18:55:31.002546 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:31.002514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2qb\" (UniqueName: \"kubernetes.io/projected/39d82901-ba40-4aed-a787-8d02b9a6db70-kube-api-access-7p2qb\") pod \"s3-init-jx5ws\" (UID: \"39d82901-ba40-4aed-a787-8d02b9a6db70\") " pod="kserve/s3-init-jx5ws"
Sep 16 18:55:31.002907 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:31.002569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d82901-ba40-4aed-a787-8d02b9a6db70-model-cache\") pod \"s3-init-jx5ws\" (UID: \"39d82901-ba40-4aed-a787-8d02b9a6db70\") " pod="kserve/s3-init-jx5ws"
Sep 16 18:55:31.002951 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:31.002928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d82901-ba40-4aed-a787-8d02b9a6db70-model-cache\") pod \"s3-init-jx5ws\" (UID: \"39d82901-ba40-4aed-a787-8d02b9a6db70\") " pod="kserve/s3-init-jx5ws"
Sep 16 18:55:31.010991 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:31.010963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2qb\" (UniqueName: \"kubernetes.io/projected/39d82901-ba40-4aed-a787-8d02b9a6db70-kube-api-access-7p2qb\") pod \"s3-init-jx5ws\" (UID: \"39d82901-ba40-4aed-a787-8d02b9a6db70\") " pod="kserve/s3-init-jx5ws"
Sep 16 18:55:31.089336 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:31.089294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jx5ws"
Sep 16 18:55:31.365367 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:31.365337    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jx5ws"]
Sep 16 18:55:31.366992 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:55:31.366964    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39d82901_ba40_4aed_a787_8d02b9a6db70.slice/crio-49b6b777dd1bee23cef110cd1c93701806ba611f93003c55036c36c85af22d8c WatchSource:0}: Error finding container 49b6b777dd1bee23cef110cd1c93701806ba611f93003c55036c36c85af22d8c: Status 404 returned error can't find the container with id 49b6b777dd1bee23cef110cd1c93701806ba611f93003c55036c36c85af22d8c
Sep 16 18:55:31.994733 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:31.994697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jx5ws" event={"ID":"39d82901-ba40-4aed-a787-8d02b9a6db70","Type":"ContainerStarted","Data":"49b6b777dd1bee23cef110cd1c93701806ba611f93003c55036c36c85af22d8c"}
Sep 16 18:55:35.010954 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:35.010919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jx5ws" event={"ID":"39d82901-ba40-4aed-a787-8d02b9a6db70","Type":"ContainerStarted","Data":"0dbfa4e46ab17e01db60f21d25c82baea8e489f61fd032c9010beceeec810cc0"}
Sep 16 18:55:37.022762 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:37.022673    2578 generic.go:356] "Generic (PLEG): container finished" podID="39d82901-ba40-4aed-a787-8d02b9a6db70" containerID="0dbfa4e46ab17e01db60f21d25c82baea8e489f61fd032c9010beceeec810cc0" exitCode=0
Sep 16 18:55:37.023212 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:37.022748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jx5ws" event={"ID":"39d82901-ba40-4aed-a787-8d02b9a6db70","Type":"ContainerDied","Data":"0dbfa4e46ab17e01db60f21d25c82baea8e489f61fd032c9010beceeec810cc0"}
Sep 16 18:55:42.052393 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:42.052354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jx5ws" event={"ID":"39d82901-ba40-4aed-a787-8d02b9a6db70","Type":"ContainerStarted","Data":"53b6afa952d9538fa94891370201d1b16ab74233d7e293182a6d470d1629400b"}
Sep 16 18:55:42.073689 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:55:42.073636    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-jx5ws" podStartSLOduration=2.370923987 podStartE2EDuration="12.073615935s" totalImagesPullingTime="8.002450235s" totalInitContainerRuntime="1.700241713s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:42.071981575 +0000 UTC m=+649.519076105" watchObservedRunningTime="2026-09-16 18:55:42.073615935 +0000 UTC m=+649.520710470"
Sep 16 18:56:47.361234 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:47.361193    2578 generic.go:356] "Generic (PLEG): container finished" podID="39d82901-ba40-4aed-a787-8d02b9a6db70" containerID="53b6afa952d9538fa94891370201d1b16ab74233d7e293182a6d470d1629400b" exitCode=0
Sep 16 18:56:47.361649 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:47.361265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jx5ws" event={"ID":"39d82901-ba40-4aed-a787-8d02b9a6db70","Type":"ContainerDied","Data":"53b6afa952d9538fa94891370201d1b16ab74233d7e293182a6d470d1629400b"}
Sep 16 18:56:48.498130 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:48.498107    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jx5ws"
Sep 16 18:56:48.580570 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:48.580531    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p2qb\" (UniqueName: \"kubernetes.io/projected/39d82901-ba40-4aed-a787-8d02b9a6db70-kube-api-access-7p2qb\") pod \"39d82901-ba40-4aed-a787-8d02b9a6db70\" (UID: \"39d82901-ba40-4aed-a787-8d02b9a6db70\") "
Sep 16 18:56:48.580776 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:48.580610    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d82901-ba40-4aed-a787-8d02b9a6db70-model-cache\") pod \"39d82901-ba40-4aed-a787-8d02b9a6db70\" (UID: \"39d82901-ba40-4aed-a787-8d02b9a6db70\") "
Sep 16 18:56:48.582647 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:48.582608    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d82901-ba40-4aed-a787-8d02b9a6db70-kube-api-access-7p2qb" (OuterVolumeSpecName: "kube-api-access-7p2qb") pod "39d82901-ba40-4aed-a787-8d02b9a6db70" (UID: "39d82901-ba40-4aed-a787-8d02b9a6db70"). InnerVolumeSpecName "kube-api-access-7p2qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:48.599976 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:48.599936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d82901-ba40-4aed-a787-8d02b9a6db70-model-cache" (OuterVolumeSpecName: "model-cache") pod "39d82901-ba40-4aed-a787-8d02b9a6db70" (UID: "39d82901-ba40-4aed-a787-8d02b9a6db70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:48.682266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:48.682182    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p2qb\" (UniqueName: \"kubernetes.io/projected/39d82901-ba40-4aed-a787-8d02b9a6db70-kube-api-access-7p2qb\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:56:48.682266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:48.682211    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39d82901-ba40-4aed-a787-8d02b9a6db70-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:56:49.378501 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:49.378388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jx5ws" event={"ID":"39d82901-ba40-4aed-a787-8d02b9a6db70","Type":"ContainerDied","Data":"49b6b777dd1bee23cef110cd1c93701806ba611f93003c55036c36c85af22d8c"}
Sep 16 18:56:49.378501 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:49.378430    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49b6b777dd1bee23cef110cd1c93701806ba611f93003c55036c36c85af22d8c"
Sep 16 18:56:49.378501 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:56:49.378434    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jx5ws"
Sep 16 18:57:10.867679 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.867639    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"]
Sep 16 18:57:10.868034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.867984    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d82901-ba40-4aed-a787-8d02b9a6db70" containerName="download-hf-model"
Sep 16 18:57:10.868034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.867995    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d82901-ba40-4aed-a787-8d02b9a6db70" containerName="download-hf-model"
Sep 16 18:57:10.868034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.868012    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d82901-ba40-4aed-a787-8d02b9a6db70" containerName="s3-init"
Sep 16 18:57:10.868034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.868017    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d82901-ba40-4aed-a787-8d02b9a6db70" containerName="s3-init"
Sep 16 18:57:10.868167 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.868080    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d82901-ba40-4aed-a787-8d02b9a6db70" containerName="s3-init"
Sep 16 18:57:10.871382 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.871363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:10.874986 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.874963    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-xgbfh\""
Sep 16 18:57:10.875121 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.874985    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:57:10.875192 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.875040    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:57:10.875256 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.875061    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:57:10.884825 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.884798    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"]
Sep 16 18:57:10.981291 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.981255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:10.981491 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.981336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-dshm\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:10.981491 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.981362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:10.981491 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.981389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-home\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:10.981491 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.981411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71df4c0-af60-43d3-b53b-ac2d350521e4-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:10.981491 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:10.981443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27w2\" (UniqueName: \"kubernetes.io/projected/c71df4c0-af60-43d3-b53b-ac2d350521e4-kube-api-access-t27w2\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.082593 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.082554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27w2\" (UniqueName: \"kubernetes.io/projected/c71df4c0-af60-43d3-b53b-ac2d350521e4-kube-api-access-t27w2\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.082793 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.082644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.082793 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.082703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-dshm\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.082793 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.082726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.082793 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.082755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-home\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.082793 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.082784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71df4c0-af60-43d3-b53b-ac2d350521e4-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.083179 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.083149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.083288 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.083162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.083288 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.083220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-home\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.085087 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.085055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-dshm\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.085296 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.085275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71df4c0-af60-43d3-b53b-ac2d350521e4-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.092684 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.092659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27w2\" (UniqueName: \"kubernetes.io/projected/c71df4c0-af60-43d3-b53b-ac2d350521e4-kube-api-access-t27w2\") pod \"fc-smoke-test-kserve-7f4687897-8npxq\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.182724 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.182629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:11.463772 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.463748    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"]
Sep 16 18:57:11.466606 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:57:11.466576    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71df4c0_af60_43d3_b53b_ac2d350521e4.slice/crio-0cb47f17da9fb597bd74ed333c10a1910393931172f56e9f528d12f5ed55c8ce WatchSource:0}: Error finding container 0cb47f17da9fb597bd74ed333c10a1910393931172f56e9f528d12f5ed55c8ce: Status 404 returned error can't find the container with id 0cb47f17da9fb597bd74ed333c10a1910393931172f56e9f528d12f5ed55c8ce
Sep 16 18:57:11.469034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.469016    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:57:11.493101 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:11.493070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq" event={"ID":"c71df4c0-af60-43d3-b53b-ac2d350521e4","Type":"ContainerStarted","Data":"0cb47f17da9fb597bd74ed333c10a1910393931172f56e9f528d12f5ed55c8ce"}
Sep 16 18:57:13.504940 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:13.504904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq" event={"ID":"c71df4c0-af60-43d3-b53b-ac2d350521e4","Type":"ContainerStarted","Data":"a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508"}
Sep 16 18:57:13.523634 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:13.523585    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq" podStartSLOduration=2.053490234 podStartE2EDuration="3.523569497s" totalImagesPullingTime="1.470079263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:13.522675089 +0000 UTC m=+740.969769639" watchObservedRunningTime="2026-09-16 18:57:13.523569497 +0000 UTC m=+740.970664027"
Sep 16 18:57:14.183440 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:14.183397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:21.183786 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:21.183743    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:21.196266 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:21.196238    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:21.560235 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:21.560204    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:57:24.776736 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.776698    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"]
Sep 16 18:57:24.780161 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.780132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:57:24.784049 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.784020    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:57:24.784186 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.784078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-b74jj\""
Sep 16 18:57:24.784257 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.784185    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:57:24.784316 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.784300    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:57:24.784376 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.784339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-54jst\""
Sep 16 18:57:24.797951 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.797883    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"]
Sep 16 18:57:24.904131 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.904092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pz6r\" (UniqueName: \"kubernetes.io/projected/22cb2e89-cb9f-496f-80a2-b028d0acff50-kube-api-access-5pz6r\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc\" (UID: \"22cb2e89-cb9f-496f-80a2-b028d0acff50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:57:24.904303 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:24.904141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cb2e89-cb9f-496f-80a2-b028d0acff50-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc\" (UID: \"22cb2e89-cb9f-496f-80a2-b028d0acff50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:57:25.004646 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:25.004598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pz6r\" (UniqueName: \"kubernetes.io/projected/22cb2e89-cb9f-496f-80a2-b028d0acff50-kube-api-access-5pz6r\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc\" (UID: \"22cb2e89-cb9f-496f-80a2-b028d0acff50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:57:25.004646 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:25.004660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cb2e89-cb9f-496f-80a2-b028d0acff50-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc\" (UID: \"22cb2e89-cb9f-496f-80a2-b028d0acff50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:57:25.007169 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:25.007142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cb2e89-cb9f-496f-80a2-b028d0acff50-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc\" (UID: \"22cb2e89-cb9f-496f-80a2-b028d0acff50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:57:25.016541 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:25.016510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pz6r\" (UniqueName: \"kubernetes.io/projected/22cb2e89-cb9f-496f-80a2-b028d0acff50-kube-api-access-5pz6r\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc\" (UID: \"22cb2e89-cb9f-496f-80a2-b028d0acff50\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:57:25.091069 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:25.091034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:57:25.364433 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:25.364289    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"]
Sep 16 18:57:25.367162 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:57:25.367132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cb2e89_cb9f_496f_80a2_b028d0acff50.slice/crio-4948eb20da66e3aa9bd1b5c881492008b5059590b4fdcad328639abc834eae13 WatchSource:0}: Error finding container 4948eb20da66e3aa9bd1b5c881492008b5059590b4fdcad328639abc834eae13: Status 404 returned error can't find the container with id 4948eb20da66e3aa9bd1b5c881492008b5059590b4fdcad328639abc834eae13
Sep 16 18:57:25.570112 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:25.570072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" event={"ID":"22cb2e89-cb9f-496f-80a2-b028d0acff50","Type":"ContainerStarted","Data":"4948eb20da66e3aa9bd1b5c881492008b5059590b4fdcad328639abc834eae13"}
Sep 16 18:57:27.582659 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:27.582614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" event={"ID":"22cb2e89-cb9f-496f-80a2-b028d0acff50","Type":"ContainerStarted","Data":"840ceb4d9ae2244fb874b18a81d59c2bf3e64c13f9a7876e59bd5d11d0b332a8"}
Sep 16 18:57:27.583034 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:27.582735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:57:27.619356 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:27.619301    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" podStartSLOduration=1.481721171 podStartE2EDuration="3.61928196s" totalImagesPullingTime="2.137560789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:27.617268371 +0000 UTC m=+755.064362927" watchObservedRunningTime="2026-09-16 18:57:27.61928196 +0000 UTC m=+755.066376491"
Sep 16 18:57:58.589783 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:57:58.589754    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 18:58:21.114612 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.114525    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"]
Sep 16 18:58:21.124034 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:58:21.123986    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:58:21.124034 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:58:21.124034 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:58:21.124034 ip-10-0-143-161 kubenswrapper[2578]:  > containerID="a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508" cmd=["/bin/sleep","15"]
Sep 16 18:58:21.124298 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.124065    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq" message="" err=<
Sep 16 18:58:21.124298 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:58:21.124298 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:58:21.124298 ip-10-0-143-161 kubenswrapper[2578]:  >
Sep 16 18:58:21.124298 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:58:21.124083    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:58:21.124298 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:58:21.124298 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:58:21.124298 ip-10-0-143-161 kubenswrapper[2578]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq" podUID="c71df4c0-af60-43d3-b53b-ac2d350521e4" containerName="main" containerID="cri-o://a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508"
Sep 16 18:58:21.124298 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.124151    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq" podUID="c71df4c0-af60-43d3-b53b-ac2d350521e4" containerName="main" containerID="cri-o://a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508" gracePeriod=60
Sep 16 18:58:21.471919 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.471890    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:58:21.607958 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.607923    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-tmp-dir\") pod \"c71df4c0-af60-43d3-b53b-ac2d350521e4\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") "
Sep 16 18:58:21.608123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.607983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-home\") pod \"c71df4c0-af60-43d3-b53b-ac2d350521e4\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") "
Sep 16 18:58:21.608123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.608031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t27w2\" (UniqueName: \"kubernetes.io/projected/c71df4c0-af60-43d3-b53b-ac2d350521e4-kube-api-access-t27w2\") pod \"c71df4c0-af60-43d3-b53b-ac2d350521e4\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") "
Sep 16 18:58:21.608123 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.608086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71df4c0-af60-43d3-b53b-ac2d350521e4-tls-certs\") pod \"c71df4c0-af60-43d3-b53b-ac2d350521e4\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") "
Sep 16 18:58:21.608247 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.608139    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-dshm\") pod \"c71df4c0-af60-43d3-b53b-ac2d350521e4\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") "
Sep 16 18:58:21.608247 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.608165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-model-cache\") pod \"c71df4c0-af60-43d3-b53b-ac2d350521e4\" (UID: \"c71df4c0-af60-43d3-b53b-ac2d350521e4\") "
Sep 16 18:58:21.608247 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.608228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c71df4c0-af60-43d3-b53b-ac2d350521e4" (UID: "c71df4c0-af60-43d3-b53b-ac2d350521e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:21.608356 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.608281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-home" (OuterVolumeSpecName: "home") pod "c71df4c0-af60-43d3-b53b-ac2d350521e4" (UID: "c71df4c0-af60-43d3-b53b-ac2d350521e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:21.608441 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.608416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "c71df4c0-af60-43d3-b53b-ac2d350521e4" (UID: "c71df4c0-af60-43d3-b53b-ac2d350521e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:21.608600 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.608516    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:58:21.608600 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.608533    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:58:21.610409 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.610378    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71df4c0-af60-43d3-b53b-ac2d350521e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c71df4c0-af60-43d3-b53b-ac2d350521e4" (UID: "c71df4c0-af60-43d3-b53b-ac2d350521e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:21.610409 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.610396    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-dshm" (OuterVolumeSpecName: "dshm") pod "c71df4c0-af60-43d3-b53b-ac2d350521e4" (UID: "c71df4c0-af60-43d3-b53b-ac2d350521e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:21.610557 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.610406    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71df4c0-af60-43d3-b53b-ac2d350521e4-kube-api-access-t27w2" (OuterVolumeSpecName: "kube-api-access-t27w2") pod "c71df4c0-af60-43d3-b53b-ac2d350521e4" (UID: "c71df4c0-af60-43d3-b53b-ac2d350521e4"). InnerVolumeSpecName "kube-api-access-t27w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:21.708972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.708892    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:58:21.708972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.708920    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71df4c0-af60-43d3-b53b-ac2d350521e4-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:58:21.708972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.708932    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t27w2\" (UniqueName: \"kubernetes.io/projected/c71df4c0-af60-43d3-b53b-ac2d350521e4-kube-api-access-t27w2\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:58:21.708972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.708941    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71df4c0-af60-43d3-b53b-ac2d350521e4-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:58:21.839609 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.839573    2578 generic.go:356] "Generic (PLEG): container finished" podID="c71df4c0-af60-43d3-b53b-ac2d350521e4" containerID="a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508" exitCode=0
Sep 16 18:58:21.839771 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.839618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq" event={"ID":"c71df4c0-af60-43d3-b53b-ac2d350521e4","Type":"ContainerDied","Data":"a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508"}
Sep 16 18:58:21.839771 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.839656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq" event={"ID":"c71df4c0-af60-43d3-b53b-ac2d350521e4","Type":"ContainerDied","Data":"0cb47f17da9fb597bd74ed333c10a1910393931172f56e9f528d12f5ed55c8ce"}
Sep 16 18:58:21.839771 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.839677    2578 scope.go:117] "RemoveContainer" containerID="a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508"
Sep 16 18:58:21.839771 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.839622    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"
Sep 16 18:58:21.848836 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.848820    2578 scope.go:117] "RemoveContainer" containerID="a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508"
Sep 16 18:58:21.849071 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:58:21.849056    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508\": container with ID starting with a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508 not found: ID does not exist" containerID="a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508"
Sep 16 18:58:21.849124 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.849078    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508"} err="failed to get container status \"a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508\": rpc error: code = NotFound desc = could not find container \"a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508\": container with ID starting with a16ef9e18a06a5750b53ba63b530afc86c0d4f45524f4af25ffd215a28d5c508 not found: ID does not exist"
Sep 16 18:58:21.861388 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.861364    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"]
Sep 16 18:58:21.865450 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:21.865426    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-8npxq"]
Sep 16 18:58:23.066332 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:58:23.066296    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71df4c0-af60-43d3-b53b-ac2d350521e4" path="/var/lib/kubelet/pods/c71df4c0-af60-43d3-b53b-ac2d350521e4/volumes"
Sep 16 18:59:16.350510 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.350460    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"]
Sep 16 18:59:16.350942 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.350820    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71df4c0-af60-43d3-b53b-ac2d350521e4" containerName="main"
Sep 16 18:59:16.350942 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.350831    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71df4c0-af60-43d3-b53b-ac2d350521e4" containerName="main"
Sep 16 18:59:16.350942 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.350905    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71df4c0-af60-43d3-b53b-ac2d350521e4" containerName="main"
Sep 16 18:59:16.353740 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.353724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.356871 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.356849    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 18:59:16.358146 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.358115    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 18:59:16.358243 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.358158    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 18:59:16.358243 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.358167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-9dmk8\""
Sep 16 18:59:16.362972 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.362950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"]
Sep 16 18:59:16.472443 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.472413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.472642 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.472454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.472642 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.472493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhvlz\" (UniqueName: \"kubernetes.io/projected/4f0a7a60-04cf-4028-8cbf-530f4f22c219-kube-api-access-hhvlz\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.472642 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.472630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.472804 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.472676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.472804 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.472762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-home\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.573508 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.573461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.573508 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.573511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhvlz\" (UniqueName: \"kubernetes.io/projected/4f0a7a60-04cf-4028-8cbf-530f4f22c219-kube-api-access-hhvlz\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.573760 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.573555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.573760 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.573575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.573760 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.573597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-home\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.573760 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.573745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.574005 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.573983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.574061 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.574027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-home\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.574061 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.574048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.575738 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.575714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.575984 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.575968    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.581595 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.581573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhvlz\" (UniqueName: \"kubernetes.io/projected/4f0a7a60-04cf-4028-8cbf-530f4f22c219-kube-api-access-hhvlz\") pod \"fc-concurrency-test-kserve-b5d44467c-hqdnc\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.664795 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.664708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:16.719780 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.719750    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"]
Sep 16 18:59:16.722687 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.722660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:16.725502 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.725469    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-krw9z\""
Sep 16 18:59:16.732781 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.732355    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"]
Sep 16 18:59:16.875238 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.875195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c439e8-aec1-4afb-a827-4ff66327be36-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb\" (UID: \"24c439e8-aec1-4afb-a827-4ff66327be36\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:16.875410 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.875245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb8zq\" (UniqueName: \"kubernetes.io/projected/24c439e8-aec1-4afb-a827-4ff66327be36-kube-api-access-vb8zq\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb\" (UID: \"24c439e8-aec1-4afb-a827-4ff66327be36\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:16.942155 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.942126    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"]
Sep 16 18:59:16.943837 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:59:16.943810    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0a7a60_04cf_4028_8cbf_530f4f22c219.slice/crio-95b37daabedfd884a95a77e9b317506ddc2bdeab6fd5848e45780a316f8ae0af WatchSource:0}: Error finding container 95b37daabedfd884a95a77e9b317506ddc2bdeab6fd5848e45780a316f8ae0af: Status 404 returned error can't find the container with id 95b37daabedfd884a95a77e9b317506ddc2bdeab6fd5848e45780a316f8ae0af
Sep 16 18:59:16.976521 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.976472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c439e8-aec1-4afb-a827-4ff66327be36-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb\" (UID: \"24c439e8-aec1-4afb-a827-4ff66327be36\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:16.976658 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.976546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb8zq\" (UniqueName: \"kubernetes.io/projected/24c439e8-aec1-4afb-a827-4ff66327be36-kube-api-access-vb8zq\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb\" (UID: \"24c439e8-aec1-4afb-a827-4ff66327be36\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:16.978888 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.978864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c439e8-aec1-4afb-a827-4ff66327be36-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb\" (UID: \"24c439e8-aec1-4afb-a827-4ff66327be36\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:16.984737 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:16.984717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb8zq\" (UniqueName: \"kubernetes.io/projected/24c439e8-aec1-4afb-a827-4ff66327be36-kube-api-access-vb8zq\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb\" (UID: \"24c439e8-aec1-4afb-a827-4ff66327be36\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:17.039215 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:17.039177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:17.111440 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:17.110834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc" event={"ID":"4f0a7a60-04cf-4028-8cbf-530f4f22c219","Type":"ContainerStarted","Data":"48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e"}
Sep 16 18:59:17.111440 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:17.110898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc" event={"ID":"4f0a7a60-04cf-4028-8cbf-530f4f22c219","Type":"ContainerStarted","Data":"95b37daabedfd884a95a77e9b317506ddc2bdeab6fd5848e45780a316f8ae0af"}
Sep 16 18:59:17.130538 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:17.130463    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc" podStartSLOduration=1.130446152 podStartE2EDuration="1.130446152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:17.129392729 +0000 UTC m=+864.576487275" watchObservedRunningTime="2026-09-16 18:59:17.130446152 +0000 UTC m=+864.577540681"
Sep 16 18:59:17.309092 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:17.309062    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"]
Sep 16 18:59:17.311076 ip-10-0-143-161 kubenswrapper[2578]: W0916 18:59:17.311044    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c439e8_aec1_4afb_a827_4ff66327be36.slice/crio-85e51190cc497a1b8d19f2acb4f398c1bc6c467f6ba13e46e567db342e2108ab WatchSource:0}: Error finding container 85e51190cc497a1b8d19f2acb4f398c1bc6c467f6ba13e46e567db342e2108ab: Status 404 returned error can't find the container with id 85e51190cc497a1b8d19f2acb4f398c1bc6c467f6ba13e46e567db342e2108ab
Sep 16 18:59:17.665172 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:17.665093    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:18.117689 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:18.117653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" event={"ID":"24c439e8-aec1-4afb-a827-4ff66327be36","Type":"ContainerStarted","Data":"a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db"}
Sep 16 18:59:18.117879 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:18.117708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" event={"ID":"24c439e8-aec1-4afb-a827-4ff66327be36","Type":"ContainerStarted","Data":"85e51190cc497a1b8d19f2acb4f398c1bc6c467f6ba13e46e567db342e2108ab"}
Sep 16 18:59:18.117879 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:18.117823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:18.135894 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:18.135846    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" podStartSLOduration=2.135827732 podStartE2EDuration="2.135827732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:18.13446886 +0000 UTC m=+865.581563416" watchObservedRunningTime="2026-09-16 18:59:18.135827732 +0000 UTC m=+865.582922261"
Sep 16 18:59:26.665822 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:26.665785    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:26.678402 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:26.678374    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:27.170932 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:27.170895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:40.256340 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:40.256306    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"]
Sep 16 18:59:48.589582 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:48.589531    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" podUID="22cb2e89-cb9f-496f-80a2-b028d0acff50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:49.123882 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:49.123846    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 18:59:53.010990 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:53.010963    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 18:59:53.012224 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:53.012205    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 18:59:53.014680 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:53.014659    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:59:53.015865 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:53.015848    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 18:59:55.276318 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.276258    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" podUID="22cb2e89-cb9f-496f-80a2-b028d0acff50" containerName="main" containerID="cri-o://840ceb4d9ae2244fb874b18a81d59c2bf3e64c13f9a7876e59bd5d11d0b332a8" gracePeriod=45
Sep 16 18:59:55.475897 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.475860    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"]
Sep 16 18:59:55.482091 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.482056    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"]
Sep 16 18:59:55.490744 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:59:55.490706    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:59:55.490744 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:55.490744 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:55.490744 ip-10-0-143-161 kubenswrapper[2578]:  > containerID="48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e" cmd=["/bin/sleep","15"]
Sep 16 18:59:55.491061 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.490761    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc" message="" err=<
Sep 16 18:59:55.491061 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:55.491061 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:55.491061 ip-10-0-143-161 kubenswrapper[2578]:  >
Sep 16 18:59:55.491061 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:59:55.490774    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:59:55.491061 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:55.491061 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:55.491061 ip-10-0-143-161 kubenswrapper[2578]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc" podUID="4f0a7a60-04cf-4028-8cbf-530f4f22c219" containerName="main" containerID="cri-o://48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e"
Sep 16 18:59:55.491061 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.490828    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc" podUID="4f0a7a60-04cf-4028-8cbf-530f4f22c219" containerName="main" containerID="cri-o://48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e" gracePeriod=60
Sep 16 18:59:55.841948 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.841923    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:55.905676 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.905592    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-model-cache\") pod \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") "
Sep 16 18:59:55.905676 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.905638    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhvlz\" (UniqueName: \"kubernetes.io/projected/4f0a7a60-04cf-4028-8cbf-530f4f22c219-kube-api-access-hhvlz\") pod \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") "
Sep 16 18:59:55.905676 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.905662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tmp-dir\") pod \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") "
Sep 16 18:59:55.905959 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.905706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-home\") pod \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") "
Sep 16 18:59:55.905959 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.905737    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tls-certs\") pod \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") "
Sep 16 18:59:55.905959 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.905756    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-dshm\") pod \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\" (UID: \"4f0a7a60-04cf-4028-8cbf-530f4f22c219\") "
Sep 16 18:59:55.905959 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.905926    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f0a7a60-04cf-4028-8cbf-530f4f22c219" (UID: "4f0a7a60-04cf-4028-8cbf-530f4f22c219"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:55.906132 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.905981    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-home" (OuterVolumeSpecName: "home") pod "4f0a7a60-04cf-4028-8cbf-530f4f22c219" (UID: "4f0a7a60-04cf-4028-8cbf-530f4f22c219"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:55.906132 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.905998    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f0a7a60-04cf-4028-8cbf-530f4f22c219" (UID: "4f0a7a60-04cf-4028-8cbf-530f4f22c219"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:55.907938 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.907910    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f0a7a60-04cf-4028-8cbf-530f4f22c219" (UID: "4f0a7a60-04cf-4028-8cbf-530f4f22c219"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:55.908047 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.907920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0a7a60-04cf-4028-8cbf-530f4f22c219-kube-api-access-hhvlz" (OuterVolumeSpecName: "kube-api-access-hhvlz") pod "4f0a7a60-04cf-4028-8cbf-530f4f22c219" (UID: "4f0a7a60-04cf-4028-8cbf-530f4f22c219"). InnerVolumeSpecName "kube-api-access-hhvlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:55.908047 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:55.907949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-dshm" (OuterVolumeSpecName: "dshm") pod "4f0a7a60-04cf-4028-8cbf-530f4f22c219" (UID: "4f0a7a60-04cf-4028-8cbf-530f4f22c219"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:56.006811 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.006776    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.006811 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.006807    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhvlz\" (UniqueName: \"kubernetes.io/projected/4f0a7a60-04cf-4028-8cbf-530f4f22c219-kube-api-access-hhvlz\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.006811 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.006817    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.007054 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.006827    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.007054 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.006835    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f0a7a60-04cf-4028-8cbf-530f4f22c219-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.007054 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.006844    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f0a7a60-04cf-4028-8cbf-530f4f22c219-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.311492 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.311448    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f0a7a60-04cf-4028-8cbf-530f4f22c219" containerID="48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e" exitCode=0
Sep 16 18:59:56.311916 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.311506    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"
Sep 16 18:59:56.311916 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.311530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc" event={"ID":"4f0a7a60-04cf-4028-8cbf-530f4f22c219","Type":"ContainerDied","Data":"48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e"}
Sep 16 18:59:56.311916 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.311584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc" event={"ID":"4f0a7a60-04cf-4028-8cbf-530f4f22c219","Type":"ContainerDied","Data":"95b37daabedfd884a95a77e9b317506ddc2bdeab6fd5848e45780a316f8ae0af"}
Sep 16 18:59:56.311916 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.311605    2578 scope.go:117] "RemoveContainer" containerID="48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e"
Sep 16 18:59:56.321085 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.321067    2578 scope.go:117] "RemoveContainer" containerID="48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e"
Sep 16 18:59:56.321353 ip-10-0-143-161 kubenswrapper[2578]: E0916 18:59:56.321335    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e\": container with ID starting with 48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e not found: ID does not exist" containerID="48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e"
Sep 16 18:59:56.321399 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.321363    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e"} err="failed to get container status \"48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e\": rpc error: code = NotFound desc = could not find container \"48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e\": container with ID starting with 48a8ae55e7771f5ba42dc1d16430e6996cee4baaab5b72e84ad1991c45c8803e not found: ID does not exist"
Sep 16 18:59:56.334957 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.334925    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"]
Sep 16 18:59:56.340459 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:56.340432    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-hqdnc"]
Sep 16 18:59:57.064991 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:57.064947    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f0a7a60-04cf-4028-8cbf-530f4f22c219" path="/var/lib/kubelet/pods/4f0a7a60-04cf-4028-8cbf-530f4f22c219/volumes"
Sep 16 18:59:58.589434 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:58.589393    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" podUID="22cb2e89-cb9f-496f-80a2-b028d0acff50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:59.124656 ip-10-0-143-161 kubenswrapper[2578]: I0916 18:59:59.124617    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:08.589046 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:08.589005    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" podUID="22cb2e89-cb9f-496f-80a2-b028d0acff50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:08.589420 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:08.589100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 19:00:09.124499 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:09.124447    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:10.496190 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:10.496130    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" containerName="main" containerID="cri-o://a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db" gracePeriod=45
Sep 16 19:00:18.589853 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:18.589810    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" podUID="22cb2e89-cb9f-496f-80a2-b028d0acff50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:19.124082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:19.124042    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:19.124247 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:19.124155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 19:00:25.459683 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:25.459638    2578 generic.go:356] "Generic (PLEG): container finished" podID="22cb2e89-cb9f-496f-80a2-b028d0acff50" containerID="840ceb4d9ae2244fb874b18a81d59c2bf3e64c13f9a7876e59bd5d11d0b332a8" exitCode=0
Sep 16 19:00:25.460036 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:25.459695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" event={"ID":"22cb2e89-cb9f-496f-80a2-b028d0acff50","Type":"ContainerDied","Data":"840ceb4d9ae2244fb874b18a81d59c2bf3e64c13f9a7876e59bd5d11d0b332a8"}
Sep 16 19:00:25.520875 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:25.520849    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 19:00:25.554668 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:25.554584    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pz6r\" (UniqueName: \"kubernetes.io/projected/22cb2e89-cb9f-496f-80a2-b028d0acff50-kube-api-access-5pz6r\") pod \"22cb2e89-cb9f-496f-80a2-b028d0acff50\" (UID: \"22cb2e89-cb9f-496f-80a2-b028d0acff50\") "
Sep 16 19:00:25.556758 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:25.556722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cb2e89-cb9f-496f-80a2-b028d0acff50-kube-api-access-5pz6r" (OuterVolumeSpecName: "kube-api-access-5pz6r") pod "22cb2e89-cb9f-496f-80a2-b028d0acff50" (UID: "22cb2e89-cb9f-496f-80a2-b028d0acff50"). InnerVolumeSpecName "kube-api-access-5pz6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:25.655688 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:25.655650    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cb2e89-cb9f-496f-80a2-b028d0acff50-tls-certs\") pod \"22cb2e89-cb9f-496f-80a2-b028d0acff50\" (UID: \"22cb2e89-cb9f-496f-80a2-b028d0acff50\") "
Sep 16 19:00:25.655896 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:25.655879    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pz6r\" (UniqueName: \"kubernetes.io/projected/22cb2e89-cb9f-496f-80a2-b028d0acff50-kube-api-access-5pz6r\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:25.657710 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:25.657685    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cb2e89-cb9f-496f-80a2-b028d0acff50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22cb2e89-cb9f-496f-80a2-b028d0acff50" (UID: "22cb2e89-cb9f-496f-80a2-b028d0acff50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:25.756263 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:25.756225    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cb2e89-cb9f-496f-80a2-b028d0acff50-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:26.466675 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:26.466645    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"
Sep 16 19:00:26.467052 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:26.466693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc" event={"ID":"22cb2e89-cb9f-496f-80a2-b028d0acff50","Type":"ContainerDied","Data":"4948eb20da66e3aa9bd1b5c881492008b5059590b4fdcad328639abc834eae13"}
Sep 16 19:00:26.467052 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:26.466761    2578 scope.go:117] "RemoveContainer" containerID="840ceb4d9ae2244fb874b18a81d59c2bf3e64c13f9a7876e59bd5d11d0b332a8"
Sep 16 19:00:26.488462 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:26.488437    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"]
Sep 16 19:00:26.492490 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:26.492448    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schepmpqc"]
Sep 16 19:00:27.063179 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:27.063138    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22cb2e89-cb9f-496f-80a2-b028d0acff50" path="/var/lib/kubelet/pods/22cb2e89-cb9f-496f-80a2-b028d0acff50/volumes"
Sep 16 19:00:29.123984 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:29.123940    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:39.124411 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:39.124370    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:40.745208 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:40.745178    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 19:00:40.785180 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:40.785141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c439e8-aec1-4afb-a827-4ff66327be36-tls-certs\") pod \"24c439e8-aec1-4afb-a827-4ff66327be36\" (UID: \"24c439e8-aec1-4afb-a827-4ff66327be36\") "
Sep 16 19:00:40.785372 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:40.785226    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb8zq\" (UniqueName: \"kubernetes.io/projected/24c439e8-aec1-4afb-a827-4ff66327be36-kube-api-access-vb8zq\") pod \"24c439e8-aec1-4afb-a827-4ff66327be36\" (UID: \"24c439e8-aec1-4afb-a827-4ff66327be36\") "
Sep 16 19:00:40.787249 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:40.787190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c439e8-aec1-4afb-a827-4ff66327be36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24c439e8-aec1-4afb-a827-4ff66327be36" (UID: "24c439e8-aec1-4afb-a827-4ff66327be36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:40.787391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:40.787304    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c439e8-aec1-4afb-a827-4ff66327be36-kube-api-access-vb8zq" (OuterVolumeSpecName: "kube-api-access-vb8zq") pod "24c439e8-aec1-4afb-a827-4ff66327be36" (UID: "24c439e8-aec1-4afb-a827-4ff66327be36"). InnerVolumeSpecName "kube-api-access-vb8zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:40.885780 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:40.885741    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb8zq\" (UniqueName: \"kubernetes.io/projected/24c439e8-aec1-4afb-a827-4ff66327be36-kube-api-access-vb8zq\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:40.885780 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:40.885770    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24c439e8-aec1-4afb-a827-4ff66327be36-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:41.539963 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:41.539929    2578 generic.go:356] "Generic (PLEG): container finished" podID="24c439e8-aec1-4afb-a827-4ff66327be36" containerID="a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db" exitCode=0
Sep 16 19:00:41.540141 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:41.539969    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"
Sep 16 19:00:41.540141 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:41.539978    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" event={"ID":"24c439e8-aec1-4afb-a827-4ff66327be36","Type":"ContainerDied","Data":"a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db"}
Sep 16 19:00:41.540141 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:41.540021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb" event={"ID":"24c439e8-aec1-4afb-a827-4ff66327be36","Type":"ContainerDied","Data":"85e51190cc497a1b8d19f2acb4f398c1bc6c467f6ba13e46e567db342e2108ab"}
Sep 16 19:00:41.540141 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:41.540042    2578 scope.go:117] "RemoveContainer" containerID="a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db"
Sep 16 19:00:41.548389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:41.548364    2578 scope.go:117] "RemoveContainer" containerID="a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db"
Sep 16 19:00:41.548688 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:00:41.548664    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db\": container with ID starting with a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db not found: ID does not exist" containerID="a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db"
Sep 16 19:00:41.548766 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:41.548695    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db"} err="failed to get container status \"a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db\": rpc error: code = NotFound desc = could not find container \"a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db\": container with ID starting with a898c751ed14a880e36e7a0e1d92a296ff8778aab493041f0b78f45c1fc991db not found: ID does not exist"
Sep 16 19:00:41.560203 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:41.560172    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"]
Sep 16 19:00:41.564022 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:41.563999    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-nrknb"]
Sep 16 19:00:43.069997 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:43.069963    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" path="/var/lib/kubelet/pods/24c439e8-aec1-4afb-a827-4ff66327be36/volumes"
Sep 16 19:00:45.862805 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.862698    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"]
Sep 16 19:00:45.863152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.863052    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22cb2e89-cb9f-496f-80a2-b028d0acff50" containerName="main"
Sep 16 19:00:45.863152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.863063    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22cb2e89-cb9f-496f-80a2-b028d0acff50" containerName="main"
Sep 16 19:00:45.863152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.863073    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0a7a60-04cf-4028-8cbf-530f4f22c219" containerName="main"
Sep 16 19:00:45.863152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.863078    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0a7a60-04cf-4028-8cbf-530f4f22c219" containerName="main"
Sep 16 19:00:45.863152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.863090    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" containerName="main"
Sep 16 19:00:45.863152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.863096    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" containerName="main"
Sep 16 19:00:45.863152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.863153    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0a7a60-04cf-4028-8cbf-530f4f22c219" containerName="main"
Sep 16 19:00:45.863559 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.863162    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22cb2e89-cb9f-496f-80a2-b028d0acff50" containerName="main"
Sep 16 19:00:45.863559 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.863169    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c439e8-aec1-4afb-a827-4ff66327be36" containerName="main"
Sep 16 19:00:45.867558 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.867536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:45.870538 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.870506    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 19:00:45.870538 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.870542    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-562tn\""
Sep 16 19:00:45.870758 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.870745    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 19:00:45.870809 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.870769    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:00:45.881728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.881704    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"]
Sep 16 19:00:45.931626 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.931587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-credential-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:45.931804 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.931635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-token\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:45.931804 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.931683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:45.931804 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.931701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-certs\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:45.931804 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.931716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57nd\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-kube-api-access-x57nd\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:45.931804 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.931733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-envoy\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:45.931804 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.931767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-data\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:45.931804 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.931784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d1c73f-7b12-427e-91ef-9302a8246e25-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:45.931804 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:45.931800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-podinfo\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.032630 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.032589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.032834 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.032640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-certs\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.032834 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.032718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57nd\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-kube-api-access-x57nd\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.032834 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.032755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-envoy\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.032834 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.032822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-data\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.033093 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.032852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d1c73f-7b12-427e-91ef-9302a8246e25-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.033093 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.032879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-podinfo\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.033093 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.032949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-credential-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.033093 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.033028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.033288 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.033106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-token\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.033288 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.033226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-certs\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.033288 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.033268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-data\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.033391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.033372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-credential-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.033743 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.033718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d1c73f-7b12-427e-91ef-9302a8246e25-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.035292 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.035274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-envoy\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.035361 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.035342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-podinfo\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.042715 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.042688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-token\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.042882 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.042864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57nd\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-kube-api-access-x57nd\") pod \"router-gateway-1-openshift-default-858fc4dc48-vgv7s\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.180099 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.179973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:46.462935 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.462876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"]
Sep 16 19:00:46.465257 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:00:46.465223    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d1c73f_7b12_427e_91ef_9302a8246e25.slice/crio-9cb64d2137b3a920954aa1d71c9fac0e4b88ef22db77aaf4ca63b596575c5674 WatchSource:0}: Error finding container 9cb64d2137b3a920954aa1d71c9fac0e4b88ef22db77aaf4ca63b596575c5674: Status 404 returned error can't find the container with id 9cb64d2137b3a920954aa1d71c9fac0e4b88ef22db77aaf4ca63b596575c5674
Sep 16 19:00:46.565707 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:46.565666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s" event={"ID":"59d1c73f-7b12-427e-91ef-9302a8246e25","Type":"ContainerStarted","Data":"9cb64d2137b3a920954aa1d71c9fac0e4b88ef22db77aaf4ca63b596575c5674"}
Sep 16 19:00:49.221578 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:49.221541    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:00:49.221816 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:49.221616    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:00:49.221816 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:49.221645    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:00:49.584671 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:49.584632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s" event={"ID":"59d1c73f-7b12-427e-91ef-9302a8246e25","Type":"ContainerStarted","Data":"66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2"}
Sep 16 19:00:49.613544 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:49.613491    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s" podStartSLOduration=1.859660387 podStartE2EDuration="4.613461321s" totalImagesPullingTime="2.753800934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:49.613322674 +0000 UTC m=+957.060417205" watchObservedRunningTime="2026-09-16 19:00:49.613461321 +0000 UTC m=+957.060555851"
Sep 16 19:00:50.181104 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:50.181065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:50.185824 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:50.185795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:50.589903 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:50.589870    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:50.590813 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:50.590793    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:00:53.337842 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.337763    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"]
Sep 16 19:00:53.342268 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.342247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.345095 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.345068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 16 19:00:53.346333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.346316    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-6wnf9\""
Sep 16 19:00:53.354343 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.354318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"]
Sep 16 19:00:53.400380 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.400342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.400549 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.400388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhtkb\" (UniqueName: \"kubernetes.io/projected/163dc937-f4dc-4512-839a-6804d9afaccc-kube-api-access-dhtkb\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.400549 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.400431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.400549 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.400473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163dc937-f4dc-4512-839a-6804d9afaccc-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.400739 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.400541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.400739 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.400572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.502011 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.501970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.502011 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.502016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtkb\" (UniqueName: \"kubernetes.io/projected/163dc937-f4dc-4512-839a-6804d9afaccc-kube-api-access-dhtkb\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.502249 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.502180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.502249 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.502209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163dc937-f4dc-4512-839a-6804d9afaccc-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.502249 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.502229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.502249 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.502247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.502442 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.502372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.502535 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.502463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.502654 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.502633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.504405 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.504381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.504700 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.504683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163dc937-f4dc-4512-839a-6804d9afaccc-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.510519 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.510498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtkb\" (UniqueName: \"kubernetes.io/projected/163dc937-f4dc-4512-839a-6804d9afaccc-kube-api-access-dhtkb\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.652751 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.652652    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:53.931871 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:53.931846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"]
Sep 16 19:00:53.934081 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:00:53.934044    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163dc937_f4dc_4512_839a_6804d9afaccc.slice/crio-758e0adc23eb09ccc324cb715817b4e2e1915e3a4834b85260a56961fef57d8c WatchSource:0}: Error finding container 758e0adc23eb09ccc324cb715817b4e2e1915e3a4834b85260a56961fef57d8c: Status 404 returned error can't find the container with id 758e0adc23eb09ccc324cb715817b4e2e1915e3a4834b85260a56961fef57d8c
Sep 16 19:00:54.609675 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:54.609633    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p" event={"ID":"163dc937-f4dc-4512-839a-6804d9afaccc","Type":"ContainerStarted","Data":"65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866"}
Sep 16 19:00:54.610071 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:54.609697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p" event={"ID":"163dc937-f4dc-4512-839a-6804d9afaccc","Type":"ContainerStarted","Data":"758e0adc23eb09ccc324cb715817b4e2e1915e3a4834b85260a56961fef57d8c"}
Sep 16 19:00:54.627376 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:54.627326    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p" podStartSLOduration=1.6273111949999999 podStartE2EDuration="1.627311195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:54.626618298 +0000 UTC m=+962.073712849" watchObservedRunningTime="2026-09-16 19:00:54.627311195 +0000 UTC m=+962.074405724"
Sep 16 19:00:54.653333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:54.653291    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:55.704033 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.704003    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"]
Sep 16 19:00:55.707411 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.707392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.710227 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.710190    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:00:55.710227 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.710197    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 19:00:55.710411 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.710209    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 19:00:55.710411 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.710255    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-7dz5r\""
Sep 16 19:00:55.727908 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.727881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"]
Sep 16 19:00:55.818265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.818228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-data\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.818463 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.818284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlvcj\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-kube-api-access-rlvcj\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.818463 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.818382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-token\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.818463 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.818430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2017ea35-24f6-44b8-8916-dec6a5eccf26-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.818627 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.818520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-certs\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.818627 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.818559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-podinfo\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.818627 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.818593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-credential-socket\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.818627 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.818625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-socket\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.818773 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.818649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-envoy\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.919707 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.919668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-token\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.919707 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.919711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2017ea35-24f6-44b8-8916-dec6a5eccf26-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.919962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.919736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-certs\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.919962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.919760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-podinfo\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.919962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.919792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-credential-socket\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.919962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.919832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-socket\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.919962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.919863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-envoy\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.919962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.919932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-data\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.920153 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.919981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlvcj\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-kube-api-access-rlvcj\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.920335 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.920312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-data\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.920515 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.920463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-certs\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.920674 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.920642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-credential-socket\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.920785 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.920693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-socket\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.920881 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.920863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2017ea35-24f6-44b8-8916-dec6a5eccf26-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.922200 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.922180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-envoy\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.922472 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.922455    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-podinfo\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.929088 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.929067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-token\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:55.929306 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:55.929283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlvcj\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-kube-api-access-rlvcj\") pod \"router-gateway-1-openshift-default-665665857b-675jl\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:56.019837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:56.019746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:56.342063 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:56.342033    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"]
Sep 16 19:00:56.343133 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:00:56.343107    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2017ea35_24f6_44b8_8916_dec6a5eccf26.slice/crio-94acc6294ed014e03c14461bede28f6960cbf59a395e12f421691c354a1dec80 WatchSource:0}: Error finding container 94acc6294ed014e03c14461bede28f6960cbf59a395e12f421691c354a1dec80: Status 404 returned error can't find the container with id 94acc6294ed014e03c14461bede28f6960cbf59a395e12f421691c354a1dec80
Sep 16 19:00:56.345405 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:56.345374    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:00:56.345502 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:56.345441    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:00:56.345502 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:56.345471    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:00:56.621257 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:56.621170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl" event={"ID":"2017ea35-24f6-44b8-8916-dec6a5eccf26","Type":"ContainerStarted","Data":"a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f"}
Sep 16 19:00:56.621257 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:56.621226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl" event={"ID":"2017ea35-24f6-44b8-8916-dec6a5eccf26","Type":"ContainerStarted","Data":"94acc6294ed014e03c14461bede28f6960cbf59a395e12f421691c354a1dec80"}
Sep 16 19:00:56.644347 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:56.644290    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl" podStartSLOduration=1.644272985 podStartE2EDuration="1.644272985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:56.642621815 +0000 UTC m=+964.089716382" watchObservedRunningTime="2026-09-16 19:00:56.644272985 +0000 UTC m=+964.091367514"
Sep 16 19:00:57.020632 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:57.020467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:57.025414 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:57.025389    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:57.625632 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:57.625604    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:57.626595 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:57.626576    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:00:58.735744 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:58.735713    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"]
Sep 16 19:00:58.744505 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:00:58.744452    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:00:58.744505 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:58.744505 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:58.744505 ip-10-0-143-161 kubenswrapper[2578]:  > containerID="65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866" cmd=["/bin/sleep","15"]
Sep 16 19:00:58.744819 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:58.744525    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p" message="" err=<
Sep 16 19:00:58.744819 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:58.744819 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:58.744819 ip-10-0-143-161 kubenswrapper[2578]:  >
Sep 16 19:00:58.744819 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:00:58.744541    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:00:58.744819 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:58.744819 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:58.744819 ip-10-0-143-161 kubenswrapper[2578]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p" podUID="163dc937-f4dc-4512-839a-6804d9afaccc" containerName="main" containerID="cri-o://65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866"
Sep 16 19:00:58.744819 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:58.744579    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p" podUID="163dc937-f4dc-4512-839a-6804d9afaccc" containerName="main" containerID="cri-o://65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866" gracePeriod=60
Sep 16 19:00:59.027685 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.027661    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:59.172251 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.171807    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-dshm\") pod \"163dc937-f4dc-4512-839a-6804d9afaccc\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") "
Sep 16 19:00:59.172618 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.172591    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-tmp-dir\") pod \"163dc937-f4dc-4512-839a-6804d9afaccc\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") "
Sep 16 19:00:59.172728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.172626    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-model-cache\") pod \"163dc937-f4dc-4512-839a-6804d9afaccc\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") "
Sep 16 19:00:59.172728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.172654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163dc937-f4dc-4512-839a-6804d9afaccc-tls-certs\") pod \"163dc937-f4dc-4512-839a-6804d9afaccc\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") "
Sep 16 19:00:59.172728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.172703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-home\") pod \"163dc937-f4dc-4512-839a-6804d9afaccc\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") "
Sep 16 19:00:59.172913 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.172771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhtkb\" (UniqueName: \"kubernetes.io/projected/163dc937-f4dc-4512-839a-6804d9afaccc-kube-api-access-dhtkb\") pod \"163dc937-f4dc-4512-839a-6804d9afaccc\" (UID: \"163dc937-f4dc-4512-839a-6804d9afaccc\") "
Sep 16 19:00:59.172913 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.172826    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "163dc937-f4dc-4512-839a-6804d9afaccc" (UID: "163dc937-f4dc-4512-839a-6804d9afaccc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:59.172913 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.172853    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-model-cache" (OuterVolumeSpecName: "model-cache") pod "163dc937-f4dc-4512-839a-6804d9afaccc" (UID: "163dc937-f4dc-4512-839a-6804d9afaccc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:59.173081 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.172957    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-home" (OuterVolumeSpecName: "home") pod "163dc937-f4dc-4512-839a-6804d9afaccc" (UID: "163dc937-f4dc-4512-839a-6804d9afaccc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:59.173081 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.173066    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:59.173081 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.173081    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:59.173223 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.173090    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:59.175029 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.175003    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-dshm" (OuterVolumeSpecName: "dshm") pod "163dc937-f4dc-4512-839a-6804d9afaccc" (UID: "163dc937-f4dc-4512-839a-6804d9afaccc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:59.175029 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.175018    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163dc937-f4dc-4512-839a-6804d9afaccc-kube-api-access-dhtkb" (OuterVolumeSpecName: "kube-api-access-dhtkb") pod "163dc937-f4dc-4512-839a-6804d9afaccc" (UID: "163dc937-f4dc-4512-839a-6804d9afaccc"). InnerVolumeSpecName "kube-api-access-dhtkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:59.175249 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.175127    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163dc937-f4dc-4512-839a-6804d9afaccc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "163dc937-f4dc-4512-839a-6804d9afaccc" (UID: "163dc937-f4dc-4512-839a-6804d9afaccc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:59.274230 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.274138    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/163dc937-f4dc-4512-839a-6804d9afaccc-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:59.274230 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.274171    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/163dc937-f4dc-4512-839a-6804d9afaccc-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:59.274230 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.274182    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhtkb\" (UniqueName: \"kubernetes.io/projected/163dc937-f4dc-4512-839a-6804d9afaccc-kube-api-access-dhtkb\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:00:59.635794 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.635757    2578 generic.go:356] "Generic (PLEG): container finished" podID="163dc937-f4dc-4512-839a-6804d9afaccc" containerID="65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866" exitCode=0
Sep 16 19:00:59.636016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.635801    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"
Sep 16 19:00:59.636016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.635854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p" event={"ID":"163dc937-f4dc-4512-839a-6804d9afaccc","Type":"ContainerDied","Data":"65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866"}
Sep 16 19:00:59.636016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.635902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p" event={"ID":"163dc937-f4dc-4512-839a-6804d9afaccc","Type":"ContainerDied","Data":"758e0adc23eb09ccc324cb715817b4e2e1915e3a4834b85260a56961fef57d8c"}
Sep 16 19:00:59.636016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.635925    2578 scope.go:117] "RemoveContainer" containerID="65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866"
Sep 16 19:00:59.645374 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.645356    2578 scope.go:117] "RemoveContainer" containerID="65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866"
Sep 16 19:00:59.645692 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:00:59.645670    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866\": container with ID starting with 65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866 not found: ID does not exist" containerID="65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866"
Sep 16 19:00:59.645791 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.645698    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866"} err="failed to get container status \"65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866\": rpc error: code = NotFound desc = could not find container \"65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866\": container with ID starting with 65f79a9203aa9dad9ab797b5ecd9aff30a4a4e90661b7fb16f3df2ad7211a866 not found: ID does not exist"
Sep 16 19:00:59.658352 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.658313    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"]
Sep 16 19:00:59.661512 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:00:59.661464    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5vl9p"]
Sep 16 19:01:00.437045 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:00.437005    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"]
Sep 16 19:01:00.437432 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:00.437209    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s" podUID="59d1c73f-7b12-427e-91ef-9302a8246e25" containerName="istio-proxy" containerID="cri-o://66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2" gracePeriod=30
Sep 16 19:01:01.064885 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:01.064851    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163dc937-f4dc-4512-839a-6804d9afaccc" path="/var/lib/kubelet/pods/163dc937-f4dc-4512-839a-6804d9afaccc/volumes"
Sep 16 19:01:03.192767 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.192740    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:01:03.314013 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.313979    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-podinfo\") pod \"59d1c73f-7b12-427e-91ef-9302a8246e25\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") "
Sep 16 19:01:03.314013 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314027    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-envoy\") pod \"59d1c73f-7b12-427e-91ef-9302a8246e25\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") "
Sep 16 19:01:03.314265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-token\") pod \"59d1c73f-7b12-427e-91ef-9302a8246e25\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") "
Sep 16 19:01:03.314265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314089    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-socket\") pod \"59d1c73f-7b12-427e-91ef-9302a8246e25\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") "
Sep 16 19:01:03.314265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-certs\") pod \"59d1c73f-7b12-427e-91ef-9302a8246e25\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") "
Sep 16 19:01:03.314265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314183    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x57nd\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-kube-api-access-x57nd\") pod \"59d1c73f-7b12-427e-91ef-9302a8246e25\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") "
Sep 16 19:01:03.314265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314213    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-data\") pod \"59d1c73f-7b12-427e-91ef-9302a8246e25\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") "
Sep 16 19:01:03.314265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d1c73f-7b12-427e-91ef-9302a8246e25-istiod-ca-cert\") pod \"59d1c73f-7b12-427e-91ef-9302a8246e25\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") "
Sep 16 19:01:03.314265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-credential-socket\") pod \"59d1c73f-7b12-427e-91ef-9302a8246e25\" (UID: \"59d1c73f-7b12-427e-91ef-9302a8246e25\") "
Sep 16 19:01:03.314640 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59d1c73f-7b12-427e-91ef-9302a8246e25" (UID: "59d1c73f-7b12-427e-91ef-9302a8246e25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:03.314640 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59d1c73f-7b12-427e-91ef-9302a8246e25" (UID: "59d1c73f-7b12-427e-91ef-9302a8246e25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:03.314640 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314551    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-data" (OuterVolumeSpecName: "istio-data") pod "59d1c73f-7b12-427e-91ef-9302a8246e25" (UID: "59d1c73f-7b12-427e-91ef-9302a8246e25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:03.314799 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59d1c73f-7b12-427e-91ef-9302a8246e25" (UID: "59d1c73f-7b12-427e-91ef-9302a8246e25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:03.314856 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.314796    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d1c73f-7b12-427e-91ef-9302a8246e25-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59d1c73f-7b12-427e-91ef-9302a8246e25" (UID: "59d1c73f-7b12-427e-91ef-9302a8246e25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:03.316408 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.316376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59d1c73f-7b12-427e-91ef-9302a8246e25" (UID: "59d1c73f-7b12-427e-91ef-9302a8246e25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:01:03.316532 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.316382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-token" (OuterVolumeSpecName: "istio-token") pod "59d1c73f-7b12-427e-91ef-9302a8246e25" (UID: "59d1c73f-7b12-427e-91ef-9302a8246e25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:03.316680 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.316659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-kube-api-access-x57nd" (OuterVolumeSpecName: "kube-api-access-x57nd") pod "59d1c73f-7b12-427e-91ef-9302a8246e25" (UID: "59d1c73f-7b12-427e-91ef-9302a8246e25"). InnerVolumeSpecName "kube-api-access-x57nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:03.316751 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.316661    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59d1c73f-7b12-427e-91ef-9302a8246e25" (UID: "59d1c73f-7b12-427e-91ef-9302a8246e25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:03.415901 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.415842    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-token\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:03.415901 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.415892    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-socket\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:03.415901 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.415903    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-workload-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:03.415901 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.415912    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x57nd\" (UniqueName: \"kubernetes.io/projected/59d1c73f-7b12-427e-91ef-9302a8246e25-kube-api-access-x57nd\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:03.415901 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.415922    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-data\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:03.416219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.415930    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d1c73f-7b12-427e-91ef-9302a8246e25-istiod-ca-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:03.416219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.415938    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-credential-socket\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:03.416219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.415945    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-podinfo\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:03.416219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.415953    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d1c73f-7b12-427e-91ef-9302a8246e25-istio-envoy\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:03.658114 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.658075    2578 generic.go:356] "Generic (PLEG): container finished" podID="59d1c73f-7b12-427e-91ef-9302a8246e25" containerID="66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2" exitCode=0
Sep 16 19:01:03.658314 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.658132    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"
Sep 16 19:01:03.658314 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.658173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s" event={"ID":"59d1c73f-7b12-427e-91ef-9302a8246e25","Type":"ContainerDied","Data":"66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2"}
Sep 16 19:01:03.658314 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.658246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s" event={"ID":"59d1c73f-7b12-427e-91ef-9302a8246e25","Type":"ContainerDied","Data":"9cb64d2137b3a920954aa1d71c9fac0e4b88ef22db77aaf4ca63b596575c5674"}
Sep 16 19:01:03.658314 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.658267    2578 scope.go:117] "RemoveContainer" containerID="66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2"
Sep 16 19:01:03.667656 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.667637    2578 scope.go:117] "RemoveContainer" containerID="66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2"
Sep 16 19:01:03.667929 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:01:03.667908    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2\": container with ID starting with 66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2 not found: ID does not exist" containerID="66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2"
Sep 16 19:01:03.667994 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.667937    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2"} err="failed to get container status \"66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2\": rpc error: code = NotFound desc = could not find container \"66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2\": container with ID starting with 66718b81e982b82bd7ce16ebb7777f032f4bdd98a5250b293892ff9f5212cbf2 not found: ID does not exist"
Sep 16 19:01:03.682578 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.682547    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"]
Sep 16 19:01:03.686584 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:03.686560    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-vgv7s"]
Sep 16 19:01:05.064201 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:05.064165    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d1c73f-7b12-427e-91ef-9302a8246e25" path="/var/lib/kubelet/pods/59d1c73f-7b12-427e-91ef-9302a8246e25/volumes"
Sep 16 19:01:10.249053 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.249017    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"]
Sep 16 19:01:10.249498 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.249362    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d1c73f-7b12-427e-91ef-9302a8246e25" containerName="istio-proxy"
Sep 16 19:01:10.249498 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.249373    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d1c73f-7b12-427e-91ef-9302a8246e25" containerName="istio-proxy"
Sep 16 19:01:10.249498 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.249395    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163dc937-f4dc-4512-839a-6804d9afaccc" containerName="main"
Sep 16 19:01:10.249498 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.249402    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="163dc937-f4dc-4512-839a-6804d9afaccc" containerName="main"
Sep 16 19:01:10.249498 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.249492    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="163dc937-f4dc-4512-839a-6804d9afaccc" containerName="main"
Sep 16 19:01:10.249498 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.249503    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d1c73f-7b12-427e-91ef-9302a8246e25" containerName="istio-proxy"
Sep 16 19:01:10.253295 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.253277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.257471 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.257437    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-cf6sn\""
Sep 16 19:01:10.257667 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.257472    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 16 19:01:10.262536 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.262510    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"]
Sep 16 19:01:10.273352 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.273323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6f7v\" (UniqueName: \"kubernetes.io/projected/5be430a5-5d60-4dc3-93cd-20ddf6292d98-kube-api-access-t6f7v\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.273539 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.273403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.273539 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.273450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.273539 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.273494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.273539 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.273518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.273761 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.273643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.374842 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.374794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.375062 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.374881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6f7v\" (UniqueName: \"kubernetes.io/projected/5be430a5-5d60-4dc3-93cd-20ddf6292d98-kube-api-access-t6f7v\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.375062 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.374927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.375062 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.374952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.375062 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.374991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.375062 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.375013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.375359 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.375335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.375433 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.375399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.375504 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.375433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.377110 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.377091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.377386 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.377367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.383548 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.383525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6f7v\" (UniqueName: \"kubernetes.io/projected/5be430a5-5d60-4dc3-93cd-20ddf6292d98-kube-api-access-t6f7v\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.565701 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.565662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:10.847893 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:10.847865    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"]
Sep 16 19:01:10.850292 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:01:10.850263    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be430a5_5d60_4dc3_93cd_20ddf6292d98.slice/crio-8ef38833019ef72fd1c284ad88fcc87b7984d4e021b458fe2c608e55c6339ba7 WatchSource:0}: Error finding container 8ef38833019ef72fd1c284ad88fcc87b7984d4e021b458fe2c608e55c6339ba7: Status 404 returned error can't find the container with id 8ef38833019ef72fd1c284ad88fcc87b7984d4e021b458fe2c608e55c6339ba7
Sep 16 19:01:11.706451 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:11.706413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf" event={"ID":"5be430a5-5d60-4dc3-93cd-20ddf6292d98","Type":"ContainerStarted","Data":"dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852"}
Sep 16 19:01:11.706861 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:11.706470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf" event={"ID":"5be430a5-5d60-4dc3-93cd-20ddf6292d98","Type":"ContainerStarted","Data":"8ef38833019ef72fd1c284ad88fcc87b7984d4e021b458fe2c608e55c6339ba7"}
Sep 16 19:01:11.726728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:11.726663    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf" podStartSLOduration=1.7266466120000001 podStartE2EDuration="1.726646612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:11.725089218 +0000 UTC m=+979.172183785" watchObservedRunningTime="2026-09-16 19:01:11.726646612 +0000 UTC m=+979.173741157"
Sep 16 19:01:12.566077 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:12.566035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:14.256716 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:14.256669    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"]
Sep 16 19:01:14.734127 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:01:14.734089    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:01:14.734127 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:14.734127 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:14.734127 ip-10-0-143-161 kubenswrapper[2578]:  > containerID="dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852" cmd=["/bin/sleep","15"]
Sep 16 19:01:14.734404 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:14.734145    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf" message="" err=<
Sep 16 19:01:14.734404 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:14.734404 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:14.734404 ip-10-0-143-161 kubenswrapper[2578]:  >
Sep 16 19:01:14.734404 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:01:14.734158    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:01:14.734404 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:14.734404 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:14.734404 ip-10-0-143-161 kubenswrapper[2578]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf" podUID="5be430a5-5d60-4dc3-93cd-20ddf6292d98" containerName="main" containerID="cri-o://dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852"
Sep 16 19:01:14.734404 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:14.734197    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf" podUID="5be430a5-5d60-4dc3-93cd-20ddf6292d98" containerName="main" containerID="cri-o://dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852" gracePeriod=60
Sep 16 19:01:15.025756 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.025731    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:15.111772 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.111696    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-model-cache\") pod \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") "
Sep 16 19:01:15.111772 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.111729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-dshm\") pod \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") "
Sep 16 19:01:15.111975 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.111794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tmp-dir\") pod \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") "
Sep 16 19:01:15.111975 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.111845    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tls-certs\") pod \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") "
Sep 16 19:01:15.111975 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.111879    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-home\") pod \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") "
Sep 16 19:01:15.111975 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.111912    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6f7v\" (UniqueName: \"kubernetes.io/projected/5be430a5-5d60-4dc3-93cd-20ddf6292d98-kube-api-access-t6f7v\") pod \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\" (UID: \"5be430a5-5d60-4dc3-93cd-20ddf6292d98\") "
Sep 16 19:01:15.112189 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.111988    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-model-cache" (OuterVolumeSpecName: "model-cache") pod "5be430a5-5d60-4dc3-93cd-20ddf6292d98" (UID: "5be430a5-5d60-4dc3-93cd-20ddf6292d98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:15.112189 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.112051    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5be430a5-5d60-4dc3-93cd-20ddf6292d98" (UID: "5be430a5-5d60-4dc3-93cd-20ddf6292d98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:15.112189 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.112080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-home" (OuterVolumeSpecName: "home") pod "5be430a5-5d60-4dc3-93cd-20ddf6292d98" (UID: "5be430a5-5d60-4dc3-93cd-20ddf6292d98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:15.112391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.112321    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:15.112391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.112336    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:15.112391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.112345    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:15.114251 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.114226    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-dshm" (OuterVolumeSpecName: "dshm") pod "5be430a5-5d60-4dc3-93cd-20ddf6292d98" (UID: "5be430a5-5d60-4dc3-93cd-20ddf6292d98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:15.114357 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.114234    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5be430a5-5d60-4dc3-93cd-20ddf6292d98" (UID: "5be430a5-5d60-4dc3-93cd-20ddf6292d98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:15.114357 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.114305    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be430a5-5d60-4dc3-93cd-20ddf6292d98-kube-api-access-t6f7v" (OuterVolumeSpecName: "kube-api-access-t6f7v") pod "5be430a5-5d60-4dc3-93cd-20ddf6292d98" (UID: "5be430a5-5d60-4dc3-93cd-20ddf6292d98"). InnerVolumeSpecName "kube-api-access-t6f7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:15.212852 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.212813    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be430a5-5d60-4dc3-93cd-20ddf6292d98-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:15.212852 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.212849    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6f7v\" (UniqueName: \"kubernetes.io/projected/5be430a5-5d60-4dc3-93cd-20ddf6292d98-kube-api-access-t6f7v\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:15.212852 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.212861    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be430a5-5d60-4dc3-93cd-20ddf6292d98-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:15.731576 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.731538    2578 generic.go:356] "Generic (PLEG): container finished" podID="5be430a5-5d60-4dc3-93cd-20ddf6292d98" containerID="dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852" exitCode=0
Sep 16 19:01:15.732065 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.731587    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"
Sep 16 19:01:15.732065 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.731628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf" event={"ID":"5be430a5-5d60-4dc3-93cd-20ddf6292d98","Type":"ContainerDied","Data":"dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852"}
Sep 16 19:01:15.732065 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.731700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf" event={"ID":"5be430a5-5d60-4dc3-93cd-20ddf6292d98","Type":"ContainerDied","Data":"8ef38833019ef72fd1c284ad88fcc87b7984d4e021b458fe2c608e55c6339ba7"}
Sep 16 19:01:15.732065 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.731738    2578 scope.go:117] "RemoveContainer" containerID="dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852"
Sep 16 19:01:15.740640 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.740445    2578 scope.go:117] "RemoveContainer" containerID="dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852"
Sep 16 19:01:15.740802 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:01:15.740784    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852\": container with ID starting with dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852 not found: ID does not exist" containerID="dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852"
Sep 16 19:01:15.740859 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.740815    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852"} err="failed to get container status \"dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852\": rpc error: code = NotFound desc = could not find container \"dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852\": container with ID starting with dd18d6e04363d9b6586446f34cb5167bb9ea61bc32e79ea206268cad2a07b852 not found: ID does not exist"
Sep 16 19:01:15.760816 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.760769    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"]
Sep 16 19:01:15.761095 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:15.761051    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfjgnbf"]
Sep 16 19:01:17.065250 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:17.065219    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be430a5-5d60-4dc3-93cd-20ddf6292d98" path="/var/lib/kubelet/pods/5be430a5-5d60-4dc3-93cd-20ddf6292d98/volumes"
Sep 16 19:01:17.763999 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:17.763959    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"]
Sep 16 19:01:17.764227 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:17.764203    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl" podUID="2017ea35-24f6-44b8-8916-dec6a5eccf26" containerName="istio-proxy" containerID="cri-o://a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f" gracePeriod=30
Sep 16 19:01:19.907498 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.907456    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:01:19.955026 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.954992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-data\") pod \"2017ea35-24f6-44b8-8916-dec6a5eccf26\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") "
Sep 16 19:01:19.955026 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2017ea35-24f6-44b8-8916-dec6a5eccf26-istiod-ca-cert\") pod \"2017ea35-24f6-44b8-8916-dec6a5eccf26\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") "
Sep 16 19:01:19.955305 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-token\") pod \"2017ea35-24f6-44b8-8916-dec6a5eccf26\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") "
Sep 16 19:01:19.955305 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-credential-socket\") pod \"2017ea35-24f6-44b8-8916-dec6a5eccf26\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") "
Sep 16 19:01:19.955305 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-certs\") pod \"2017ea35-24f6-44b8-8916-dec6a5eccf26\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") "
Sep 16 19:01:19.955305 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-socket\") pod \"2017ea35-24f6-44b8-8916-dec6a5eccf26\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") "
Sep 16 19:01:19.955305 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-envoy\") pod \"2017ea35-24f6-44b8-8916-dec6a5eccf26\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") "
Sep 16 19:01:19.955305 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlvcj\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-kube-api-access-rlvcj\") pod \"2017ea35-24f6-44b8-8916-dec6a5eccf26\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") "
Sep 16 19:01:19.955688 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-podinfo\") pod \"2017ea35-24f6-44b8-8916-dec6a5eccf26\" (UID: \"2017ea35-24f6-44b8-8916-dec6a5eccf26\") "
Sep 16 19:01:19.955688 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955319    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-data" (OuterVolumeSpecName: "istio-data") pod "2017ea35-24f6-44b8-8916-dec6a5eccf26" (UID: "2017ea35-24f6-44b8-8916-dec6a5eccf26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:19.955688 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955554    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2017ea35-24f6-44b8-8916-dec6a5eccf26" (UID: "2017ea35-24f6-44b8-8916-dec6a5eccf26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:19.955688 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2017ea35-24f6-44b8-8916-dec6a5eccf26-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2017ea35-24f6-44b8-8916-dec6a5eccf26" (UID: "2017ea35-24f6-44b8-8916-dec6a5eccf26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:19.955912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2017ea35-24f6-44b8-8916-dec6a5eccf26" (UID: "2017ea35-24f6-44b8-8916-dec6a5eccf26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:19.955912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.955767    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2017ea35-24f6-44b8-8916-dec6a5eccf26" (UID: "2017ea35-24f6-44b8-8916-dec6a5eccf26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:19.958060 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.957875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-token" (OuterVolumeSpecName: "istio-token") pod "2017ea35-24f6-44b8-8916-dec6a5eccf26" (UID: "2017ea35-24f6-44b8-8916-dec6a5eccf26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:19.958060 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.957950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-kube-api-access-rlvcj" (OuterVolumeSpecName: "kube-api-access-rlvcj") pod "2017ea35-24f6-44b8-8916-dec6a5eccf26" (UID: "2017ea35-24f6-44b8-8916-dec6a5eccf26"). InnerVolumeSpecName "kube-api-access-rlvcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:19.958060 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.957983    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2017ea35-24f6-44b8-8916-dec6a5eccf26" (UID: "2017ea35-24f6-44b8-8916-dec6a5eccf26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:01:19.958288 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:19.958271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2017ea35-24f6-44b8-8916-dec6a5eccf26" (UID: "2017ea35-24f6-44b8-8916-dec6a5eccf26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:20.056878 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.056839    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-data\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.056878 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.056871    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2017ea35-24f6-44b8-8916-dec6a5eccf26-istiod-ca-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.056878 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.056880    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-token\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.056878 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.056888    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-credential-socket\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.057148 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.056897    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.057148 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.056905    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-workload-socket\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.057148 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.056913    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-envoy\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.057148 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.056921    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlvcj\" (UniqueName: \"kubernetes.io/projected/2017ea35-24f6-44b8-8916-dec6a5eccf26-kube-api-access-rlvcj\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.057148 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.056929    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2017ea35-24f6-44b8-8916-dec6a5eccf26-istio-podinfo\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:01:20.757504 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.757447    2578 generic.go:356] "Generic (PLEG): container finished" podID="2017ea35-24f6-44b8-8916-dec6a5eccf26" containerID="a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f" exitCode=0
Sep 16 19:01:20.757711 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.757513    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"
Sep 16 19:01:20.757711 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.757532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl" event={"ID":"2017ea35-24f6-44b8-8916-dec6a5eccf26","Type":"ContainerDied","Data":"a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f"}
Sep 16 19:01:20.757711 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.757588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl" event={"ID":"2017ea35-24f6-44b8-8916-dec6a5eccf26","Type":"ContainerDied","Data":"94acc6294ed014e03c14461bede28f6960cbf59a395e12f421691c354a1dec80"}
Sep 16 19:01:20.757711 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.757613    2578 scope.go:117] "RemoveContainer" containerID="a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f"
Sep 16 19:01:20.767096 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.767077    2578 scope.go:117] "RemoveContainer" containerID="a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f"
Sep 16 19:01:20.767402 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:01:20.767383    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f\": container with ID starting with a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f not found: ID does not exist" containerID="a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f"
Sep 16 19:01:20.767509 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.767411    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f"} err="failed to get container status \"a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f\": rpc error: code = NotFound desc = could not find container \"a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f\": container with ID starting with a838c084c4cadf1536401c083581c8877d9f0ec75cb9e14e25bfc3175d55504f not found: ID does not exist"
Sep 16 19:01:20.782201 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.782174    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"]
Sep 16 19:01:20.787275 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:20.787251    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-675jl"]
Sep 16 19:01:21.064175 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:21.064138    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2017ea35-24f6-44b8-8916-dec6a5eccf26" path="/var/lib/kubelet/pods/2017ea35-24f6-44b8-8916-dec6a5eccf26/volumes"
Sep 16 19:01:25.353589 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.353543    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"]
Sep 16 19:01:25.354023 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.353979    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2017ea35-24f6-44b8-8916-dec6a5eccf26" containerName="istio-proxy"
Sep 16 19:01:25.354023 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.353994    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2017ea35-24f6-44b8-8916-dec6a5eccf26" containerName="istio-proxy"
Sep 16 19:01:25.354023 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.354005    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be430a5-5d60-4dc3-93cd-20ddf6292d98" containerName="main"
Sep 16 19:01:25.354023 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.354011    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be430a5-5d60-4dc3-93cd-20ddf6292d98" containerName="main"
Sep 16 19:01:25.354156 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.354068    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2017ea35-24f6-44b8-8916-dec6a5eccf26" containerName="istio-proxy"
Sep 16 19:01:25.354156 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.354075    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be430a5-5d60-4dc3-93cd-20ddf6292d98" containerName="main"
Sep 16 19:01:25.357188 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.357168    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.359895 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.359862    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 19:01:25.360040 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.359917    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 19:01:25.360040 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.359994    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:01:25.360040 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.360031    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 19:01:25.360228 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.360153    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-vplvx\""
Sep 16 19:01:25.361261 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.361234    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 19:01:25.368539 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.368518    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"]
Sep 16 19:01:25.397199 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.397167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-tmp-dir\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.397357 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.397203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69sxx\" (UniqueName: \"kubernetes.io/projected/a253fabb-5e64-456f-8997-d286b067ca44-kube-api-access-69sxx\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.397357 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.397233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a253fabb-5e64-456f-8997-d286b067ca44-cabundle-cert\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.397357 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.397285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-model-cache\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.397506 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.397361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-kserve-provision-location\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.397506 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.397387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-dshm\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.397506 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.397436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-home\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.397506 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.397455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a253fabb-5e64-456f-8997-d286b067ca44-tls-certs\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498032 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.497990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a253fabb-5e64-456f-8997-d286b067ca44-cabundle-cert\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498032 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-model-cache\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498297 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-kserve-provision-location\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498297 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-dshm\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498297 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-home\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498297 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a253fabb-5e64-456f-8997-d286b067ca44-tls-certs\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498547 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-tmp-dir\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498547 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69sxx\" (UniqueName: \"kubernetes.io/projected/a253fabb-5e64-456f-8997-d286b067ca44-kube-api-access-69sxx\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498666 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-model-cache\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498666 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-kserve-provision-location\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498666 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-home\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498863 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-tmp-dir\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.498918 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.498865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a253fabb-5e64-456f-8997-d286b067ca44-cabundle-cert\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.500665 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.500644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-dshm\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.500856 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.500843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a253fabb-5e64-456f-8997-d286b067ca44-tls-certs\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.506787 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.506770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69sxx\" (UniqueName: \"kubernetes.io/projected/a253fabb-5e64-456f-8997-d286b067ca44-kube-api-access-69sxx\") pod \"auth-enabled-test-kserve-57954458c5-x95cz\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.668846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.668422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:25.670659 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.670631    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"]
Sep 16 19:01:25.673203 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.673183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:01:25.675827 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.675798    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-hf9bl\""
Sep 16 19:01:25.682689 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.682664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"]
Sep 16 19:01:25.699698 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.699397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9734371d-f0c7-4346-9502-f4e9afff7953-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj\" (UID: \"9734371d-f0c7-4346-9502-f4e9afff7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:01:25.699698 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.699453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc49n\" (UniqueName: \"kubernetes.io/projected/9734371d-f0c7-4346-9502-f4e9afff7953-kube-api-access-zc49n\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj\" (UID: \"9734371d-f0c7-4346-9502-f4e9afff7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:01:25.800726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.800697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9734371d-f0c7-4346-9502-f4e9afff7953-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj\" (UID: \"9734371d-f0c7-4346-9502-f4e9afff7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:01:25.800918 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.800731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc49n\" (UniqueName: \"kubernetes.io/projected/9734371d-f0c7-4346-9502-f4e9afff7953-kube-api-access-zc49n\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj\" (UID: \"9734371d-f0c7-4346-9502-f4e9afff7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:01:25.803200 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.803178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9734371d-f0c7-4346-9502-f4e9afff7953-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj\" (UID: \"9734371d-f0c7-4346-9502-f4e9afff7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:01:25.808909 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.808885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc49n\" (UniqueName: \"kubernetes.io/projected/9734371d-f0c7-4346-9502-f4e9afff7953-kube-api-access-zc49n\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj\" (UID: \"9734371d-f0c7-4346-9502-f4e9afff7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:01:25.950658 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:25.950456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"]
Sep 16 19:01:25.953380 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:01:25.953352    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda253fabb_5e64_456f_8997_d286b067ca44.slice/crio-b05a5ff5690b1e7302519f04beacaa9a03be64261f526bdb788139df75b71c00 WatchSource:0}: Error finding container b05a5ff5690b1e7302519f04beacaa9a03be64261f526bdb788139df75b71c00: Status 404 returned error can't find the container with id b05a5ff5690b1e7302519f04beacaa9a03be64261f526bdb788139df75b71c00
Sep 16 19:01:26.000813 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:26.000786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:01:26.275461 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:26.275413    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"]
Sep 16 19:01:26.277603 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:01:26.277568    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9734371d_f0c7_4346_9502_f4e9afff7953.slice/crio-6a51a172b7c64a0b34baf3c93de18ab5076eb63f7fa838c658cee611f6023ecf WatchSource:0}: Error finding container 6a51a172b7c64a0b34baf3c93de18ab5076eb63f7fa838c658cee611f6023ecf: Status 404 returned error can't find the container with id 6a51a172b7c64a0b34baf3c93de18ab5076eb63f7fa838c658cee611f6023ecf
Sep 16 19:01:26.795833 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:26.795790    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" event={"ID":"a253fabb-5e64-456f-8997-d286b067ca44","Type":"ContainerStarted","Data":"391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19"}
Sep 16 19:01:26.796320 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:26.795855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" event={"ID":"a253fabb-5e64-456f-8997-d286b067ca44","Type":"ContainerStarted","Data":"b05a5ff5690b1e7302519f04beacaa9a03be64261f526bdb788139df75b71c00"}
Sep 16 19:01:26.797359 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:26.797329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" event={"ID":"9734371d-f0c7-4346-9502-f4e9afff7953","Type":"ContainerStarted","Data":"3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c"}
Sep 16 19:01:26.797523 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:26.797378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" event={"ID":"9734371d-f0c7-4346-9502-f4e9afff7953","Type":"ContainerStarted","Data":"6a51a172b7c64a0b34baf3c93de18ab5076eb63f7fa838c658cee611f6023ecf"}
Sep 16 19:01:26.797523 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:26.797447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:01:26.839317 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:26.839244    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" podStartSLOduration=1.8392203999999999 podStartE2EDuration="1.8392204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:26.838061544 +0000 UTC m=+994.285156093" watchObservedRunningTime="2026-09-16 19:01:26.8392204 +0000 UTC m=+994.286314931"
Sep 16 19:01:28.809428 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:28.809394    2578 generic.go:356] "Generic (PLEG): container finished" podID="a253fabb-5e64-456f-8997-d286b067ca44" containerID="391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19" exitCode=0
Sep 16 19:01:28.809939 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:28.809470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" event={"ID":"a253fabb-5e64-456f-8997-d286b067ca44","Type":"ContainerDied","Data":"391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19"}
Sep 16 19:01:56.982544 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:56.982509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" event={"ID":"a253fabb-5e64-456f-8997-d286b067ca44","Type":"ContainerStarted","Data":"5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78"}
Sep 16 19:01:57.008738 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:57.008645    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podStartSLOduration=1.7975751089999998 podStartE2EDuration="32.008622908s" totalImagesPullingTime="28.010343593s" totalInitContainerRuntime="2.200704206s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:57.005858632 +0000 UTC m=+1024.452953177" watchObservedRunningTime="2026-09-16 19:01:57.008622908 +0000 UTC m=+1024.455717439"
Sep 16 19:01:57.668769 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:57.668728    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:01:57.805717 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:01:57.805686    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:02:05.668873 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:02:05.668823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:02:05.670512 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:02:05.670466    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:02:15.669386 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:02:15.669340    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:02:25.669238 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:02:25.669186    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:02:35.669923 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:02:35.669879    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:02:45.669736 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:02:45.669629    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:02:55.669794 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:02:55.669752    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:03:05.669356 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:03:05.669314    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:03:15.669299 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:03:15.669244    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:03:25.669499 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:03:25.669437    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:03:35.669423 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:03:35.669372    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:03:45.669802 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:03:45.669752    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:03:55.679103 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:03:55.679069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:03:55.686975 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:03:55.686950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:04:10.524404 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:10.524370    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"]
Sep 16 19:04:10.538084 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:10.538051    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"]
Sep 16 19:04:17.805672 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:17.805586    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" podUID="9734371d-f0c7-4346-9502-f4e9afff7953" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:25.545303 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:25.545243    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" podUID="9734371d-f0c7-4346-9502-f4e9afff7953" containerName="main" containerID="cri-o://3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c" gracePeriod=45
Sep 16 19:04:25.560865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:25.560814    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" containerID="cri-o://5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78" gracePeriod=45
Sep 16 19:04:25.679844 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:25.679801    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 19:04:26.309394 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.309368    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:04:26.412240 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412161    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69sxx\" (UniqueName: \"kubernetes.io/projected/a253fabb-5e64-456f-8997-d286b067ca44-kube-api-access-69sxx\") pod \"a253fabb-5e64-456f-8997-d286b067ca44\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") "
Sep 16 19:04:26.412240 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-model-cache\") pod \"a253fabb-5e64-456f-8997-d286b067ca44\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") "
Sep 16 19:04:26.412240 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412236    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-kserve-provision-location\") pod \"a253fabb-5e64-456f-8997-d286b067ca44\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") "
Sep 16 19:04:26.412530 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412252    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a253fabb-5e64-456f-8997-d286b067ca44-tls-certs\") pod \"a253fabb-5e64-456f-8997-d286b067ca44\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") "
Sep 16 19:04:26.412530 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412303    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a253fabb-5e64-456f-8997-d286b067ca44-cabundle-cert\") pod \"a253fabb-5e64-456f-8997-d286b067ca44\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") "
Sep 16 19:04:26.412530 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412330    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-tmp-dir\") pod \"a253fabb-5e64-456f-8997-d286b067ca44\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") "
Sep 16 19:04:26.412530 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412452    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-dshm\") pod \"a253fabb-5e64-456f-8997-d286b067ca44\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") "
Sep 16 19:04:26.412752 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-model-cache" (OuterVolumeSpecName: "model-cache") pod "a253fabb-5e64-456f-8997-d286b067ca44" (UID: "a253fabb-5e64-456f-8997-d286b067ca44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:26.412752 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412558    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-home\") pod \"a253fabb-5e64-456f-8997-d286b067ca44\" (UID: \"a253fabb-5e64-456f-8997-d286b067ca44\") "
Sep 16 19:04:26.412861 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.412838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a253fabb-5e64-456f-8997-d286b067ca44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a253fabb-5e64-456f-8997-d286b067ca44" (UID: "a253fabb-5e64-456f-8997-d286b067ca44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:26.413119 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.413096    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a253fabb-5e64-456f-8997-d286b067ca44-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.413215 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.413124    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.413215 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.413172    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-home" (OuterVolumeSpecName: "home") pod "a253fabb-5e64-456f-8997-d286b067ca44" (UID: "a253fabb-5e64-456f-8997-d286b067ca44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:26.414803 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.414772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-dshm" (OuterVolumeSpecName: "dshm") pod "a253fabb-5e64-456f-8997-d286b067ca44" (UID: "a253fabb-5e64-456f-8997-d286b067ca44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:26.414951 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.414927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a253fabb-5e64-456f-8997-d286b067ca44-kube-api-access-69sxx" (OuterVolumeSpecName: "kube-api-access-69sxx") pod "a253fabb-5e64-456f-8997-d286b067ca44" (UID: "a253fabb-5e64-456f-8997-d286b067ca44"). InnerVolumeSpecName "kube-api-access-69sxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:26.415146 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.415128    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a253fabb-5e64-456f-8997-d286b067ca44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a253fabb-5e64-456f-8997-d286b067ca44" (UID: "a253fabb-5e64-456f-8997-d286b067ca44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:26.425244 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.425212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a253fabb-5e64-456f-8997-d286b067ca44" (UID: "a253fabb-5e64-456f-8997-d286b067ca44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:26.432773 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.432743    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a253fabb-5e64-456f-8997-d286b067ca44" (UID: "a253fabb-5e64-456f-8997-d286b067ca44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:26.513883 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.513849    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.513883 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.513880    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.513883 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.513889    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.514115 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.513897    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69sxx\" (UniqueName: \"kubernetes.io/projected/a253fabb-5e64-456f-8997-d286b067ca44-kube-api-access-69sxx\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.514115 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.513906    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a253fabb-5e64-456f-8997-d286b067ca44-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.514115 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.513918    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a253fabb-5e64-456f-8997-d286b067ca44-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:26.741547 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.741441    2578 generic.go:356] "Generic (PLEG): container finished" podID="a253fabb-5e64-456f-8997-d286b067ca44" containerID="5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78" exitCode=0
Sep 16 19:04:26.741547 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.741500    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"
Sep 16 19:04:26.741547 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.741526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" event={"ID":"a253fabb-5e64-456f-8997-d286b067ca44","Type":"ContainerDied","Data":"5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78"}
Sep 16 19:04:26.742074 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.741584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz" event={"ID":"a253fabb-5e64-456f-8997-d286b067ca44","Type":"ContainerDied","Data":"b05a5ff5690b1e7302519f04beacaa9a03be64261f526bdb788139df75b71c00"}
Sep 16 19:04:26.742074 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.741608    2578 scope.go:117] "RemoveContainer" containerID="5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78"
Sep 16 19:04:26.750621 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.750600    2578 scope.go:117] "RemoveContainer" containerID="391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19"
Sep 16 19:04:26.758722 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.758703    2578 scope.go:117] "RemoveContainer" containerID="5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78"
Sep 16 19:04:26.759001 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:04:26.758982    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78\": container with ID starting with 5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78 not found: ID does not exist" containerID="5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78"
Sep 16 19:04:26.759085 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.759015    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78"} err="failed to get container status \"5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78\": rpc error: code = NotFound desc = could not find container \"5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78\": container with ID starting with 5b7355097f2488b605d3b823ee50c9680f079050de7ea6d4a37a4e3f40908e78 not found: ID does not exist"
Sep 16 19:04:26.759085 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.759042    2578 scope.go:117] "RemoveContainer" containerID="391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19"
Sep 16 19:04:26.759313 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:04:26.759296    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19\": container with ID starting with 391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19 not found: ID does not exist" containerID="391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19"
Sep 16 19:04:26.759354 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.759321    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19"} err="failed to get container status \"391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19\": rpc error: code = NotFound desc = could not find container \"391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19\": container with ID starting with 391aadf8c8c54cd58b3a12bbfa8129be5a8149e7908a1839c6fce31f13bb3f19 not found: ID does not exist"
Sep 16 19:04:26.773565 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.773542    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"]
Sep 16 19:04:26.780015 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:26.779994    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-57954458c5-x95cz"]
Sep 16 19:04:27.067687 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:27.065685    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a253fabb-5e64-456f-8997-d286b067ca44" path="/var/lib/kubelet/pods/a253fabb-5e64-456f-8997-d286b067ca44/volumes"
Sep 16 19:04:27.805652 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:27.805618    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" podUID="9734371d-f0c7-4346-9502-f4e9afff7953" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:37.805452 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:37.805411    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" podUID="9734371d-f0c7-4346-9502-f4e9afff7953" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:37.805854 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:37.805539    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:04:47.805618 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:47.805567    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" podUID="9734371d-f0c7-4346-9502-f4e9afff7953" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:53.040340 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:53.040307    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:04:53.042887 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:53.042864    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:04:53.044239 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:53.044218    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:04:53.046798 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:53.046778    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:04:55.797943 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.797915    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:04:55.864923 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.864823    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc49n\" (UniqueName: \"kubernetes.io/projected/9734371d-f0c7-4346-9502-f4e9afff7953-kube-api-access-zc49n\") pod \"9734371d-f0c7-4346-9502-f4e9afff7953\" (UID: \"9734371d-f0c7-4346-9502-f4e9afff7953\") "
Sep 16 19:04:55.864923 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.864869    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9734371d-f0c7-4346-9502-f4e9afff7953-tls-certs\") pod \"9734371d-f0c7-4346-9502-f4e9afff7953\" (UID: \"9734371d-f0c7-4346-9502-f4e9afff7953\") "
Sep 16 19:04:55.867015 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.866986    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9734371d-f0c7-4346-9502-f4e9afff7953-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9734371d-f0c7-4346-9502-f4e9afff7953" (UID: "9734371d-f0c7-4346-9502-f4e9afff7953"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:55.867149 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.867056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9734371d-f0c7-4346-9502-f4e9afff7953-kube-api-access-zc49n" (OuterVolumeSpecName: "kube-api-access-zc49n") pod "9734371d-f0c7-4346-9502-f4e9afff7953" (UID: "9734371d-f0c7-4346-9502-f4e9afff7953"). InnerVolumeSpecName "kube-api-access-zc49n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:55.882519 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.882469    2578 generic.go:356] "Generic (PLEG): container finished" podID="9734371d-f0c7-4346-9502-f4e9afff7953" containerID="3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c" exitCode=0
Sep 16 19:04:55.882651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.882528    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"
Sep 16 19:04:55.882651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.882573    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" event={"ID":"9734371d-f0c7-4346-9502-f4e9afff7953","Type":"ContainerDied","Data":"3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c"}
Sep 16 19:04:55.882651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.882623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj" event={"ID":"9734371d-f0c7-4346-9502-f4e9afff7953","Type":"ContainerDied","Data":"6a51a172b7c64a0b34baf3c93de18ab5076eb63f7fa838c658cee611f6023ecf"}
Sep 16 19:04:55.882651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.882647    2578 scope.go:117] "RemoveContainer" containerID="3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c"
Sep 16 19:04:55.891959 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.891940    2578 scope.go:117] "RemoveContainer" containerID="3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c"
Sep 16 19:04:55.892234 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:04:55.892214    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c\": container with ID starting with 3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c not found: ID does not exist" containerID="3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c"
Sep 16 19:04:55.892277 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.892243    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c"} err="failed to get container status \"3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c\": rpc error: code = NotFound desc = could not find container \"3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c\": container with ID starting with 3b51d6ba7ac7ccb9972803c481b63ba16a7a76fffcdaacce663c641441dcb60c not found: ID does not exist"
Sep 16 19:04:55.906017 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.905988    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"]
Sep 16 19:04:55.910514 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.910473    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-7vrgj"]
Sep 16 19:04:55.965917 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.965886    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc49n\" (UniqueName: \"kubernetes.io/projected/9734371d-f0c7-4346-9502-f4e9afff7953-kube-api-access-zc49n\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:55.965917 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:55.965912    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9734371d-f0c7-4346-9502-f4e9afff7953-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:04:57.064316 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:04:57.064276    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9734371d-f0c7-4346-9502-f4e9afff7953" path="/var/lib/kubelet/pods/9734371d-f0c7-4346-9502-f4e9afff7953/volumes"
Sep 16 19:05:07.663792 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.663732    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"]
Sep 16 19:05:07.664312 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.664216    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9734371d-f0c7-4346-9502-f4e9afff7953" containerName="main"
Sep 16 19:05:07.664312 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.664237    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9734371d-f0c7-4346-9502-f4e9afff7953" containerName="main"
Sep 16 19:05:07.664312 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.664252    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="storage-initializer"
Sep 16 19:05:07.664312 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.664260    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="storage-initializer"
Sep 16 19:05:07.664312 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.664271    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main"
Sep 16 19:05:07.664312 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.664280    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main"
Sep 16 19:05:07.664638 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.664389    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a253fabb-5e64-456f-8997-d286b067ca44" containerName="main"
Sep 16 19:05:07.664638 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.664403    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9734371d-f0c7-4346-9502-f4e9afff7953" containerName="main"
Sep 16 19:05:07.669034 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.669014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.672345 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.672321    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 19:05:07.672598 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.672575    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:05:07.672731 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.672685    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-tbw9w\""
Sep 16 19:05:07.672731 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.672687    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 19:05:07.673725 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.673644    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 19:05:07.673725 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.673660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 19:05:07.680213 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.680189    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"]
Sep 16 19:05:07.766931 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.766890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d73b7-9bf0-486b-a339-762203c7c717-tls-certs\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.767110 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.766956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-dshm\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.767110 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.766989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d73b7-9bf0-486b-a339-762203c7c717-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.767110 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.767056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.767110 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.767090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-tmp-dir\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.767258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.767136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-model-cache\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.767258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.767166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-home\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.767258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.767194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mhf\" (UniqueName: \"kubernetes.io/projected/be8d73b7-9bf0-486b-a339-762203c7c717-kube-api-access-j8mhf\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868218 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-home\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mhf\" (UniqueName: \"kubernetes.io/projected/be8d73b7-9bf0-486b-a339-762203c7c717-kube-api-access-j8mhf\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d73b7-9bf0-486b-a339-762203c7c717-tls-certs\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-dshm\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d73b7-9bf0-486b-a339-762203c7c717-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-tmp-dir\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-model-cache\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868838 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-home\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868838 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868953 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-model-cache\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.868953 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.868924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-tmp-dir\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.869077 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.869062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d73b7-9bf0-486b-a339-762203c7c717-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.870702 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.870683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-dshm\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.870856 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.870841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d73b7-9bf0-486b-a339-762203c7c717-tls-certs\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.878448 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.878424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mhf\" (UniqueName: \"kubernetes.io/projected/be8d73b7-9bf0-486b-a339-762203c7c717-kube-api-access-j8mhf\") pod \"auth-invalid-token-test-kserve-8677dbd987-vgc5q\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:07.980990 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:07.980887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:08.265615 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:08.265586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"]
Sep 16 19:05:08.267768 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:05:08.267736    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8d73b7_9bf0_486b_a339_762203c7c717.slice/crio-35e9b8b6353e2a523c8606f9c14064ad686ff73c2ba227530d01e9cc4cbf94f2 WatchSource:0}: Error finding container 35e9b8b6353e2a523c8606f9c14064ad686ff73c2ba227530d01e9cc4cbf94f2: Status 404 returned error can't find the container with id 35e9b8b6353e2a523c8606f9c14064ad686ff73c2ba227530d01e9cc4cbf94f2
Sep 16 19:05:08.944468 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:08.944427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" event={"ID":"be8d73b7-9bf0-486b-a339-762203c7c717","Type":"ContainerStarted","Data":"ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624"}
Sep 16 19:05:08.944875 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:08.944535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" event={"ID":"be8d73b7-9bf0-486b-a339-762203c7c717","Type":"ContainerStarted","Data":"35e9b8b6353e2a523c8606f9c14064ad686ff73c2ba227530d01e9cc4cbf94f2"}
Sep 16 19:05:10.955769 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:10.955738    2578 generic.go:356] "Generic (PLEG): container finished" podID="be8d73b7-9bf0-486b-a339-762203c7c717" containerID="ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624" exitCode=0
Sep 16 19:05:10.956155 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:10.955798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" event={"ID":"be8d73b7-9bf0-486b-a339-762203c7c717","Type":"ContainerDied","Data":"ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624"}
Sep 16 19:05:11.963389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:11.963348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" event={"ID":"be8d73b7-9bf0-486b-a339-762203c7c717","Type":"ContainerStarted","Data":"7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e"}
Sep 16 19:05:11.981953 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:11.981921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:11.994152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:11.994094    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podStartSLOduration=2.823075478 podStartE2EDuration="4.994076367s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.171000889s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:11.991431997 +0000 UTC m=+1219.438526527" watchObservedRunningTime="2026-09-16 19:05:11.994076367 +0000 UTC m=+1219.441170897"
Sep 16 19:05:17.981069 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:17.981020    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:05:17.982731 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:17.982698    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:05:27.981969 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:27.981922    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:05:32.320809 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.320775    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"]
Sep 16 19:05:32.323398 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.323373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:05:32.327889 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.327684    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-qfhxz\""
Sep 16 19:05:32.327889 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.327693    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:05:32.327889 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.327736    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:05:32.327889 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.327690    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-5lnfz\""
Sep 16 19:05:32.328165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.328017    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:05:32.333878 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.333853    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"]
Sep 16 19:05:32.485380 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.485337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8a563e-4735-476c-a831-c840508be692-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb\" (UID: \"ee8a563e-4735-476c-a831-c840508be692\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:05:32.485590 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.485445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69gp\" (UniqueName: \"kubernetes.io/projected/ee8a563e-4735-476c-a831-c840508be692-kube-api-access-k69gp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb\" (UID: \"ee8a563e-4735-476c-a831-c840508be692\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:05:32.586011 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.585902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69gp\" (UniqueName: \"kubernetes.io/projected/ee8a563e-4735-476c-a831-c840508be692-kube-api-access-k69gp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb\" (UID: \"ee8a563e-4735-476c-a831-c840508be692\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:05:32.586011 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.585975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8a563e-4735-476c-a831-c840508be692-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb\" (UID: \"ee8a563e-4735-476c-a831-c840508be692\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:05:32.588404 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.588376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8a563e-4735-476c-a831-c840508be692-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb\" (UID: \"ee8a563e-4735-476c-a831-c840508be692\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:05:32.600666 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.600640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69gp\" (UniqueName: \"kubernetes.io/projected/ee8a563e-4735-476c-a831-c840508be692-kube-api-access-k69gp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb\" (UID: \"ee8a563e-4735-476c-a831-c840508be692\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:05:32.635068 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.635025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:05:32.910250 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:32.910216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"]
Sep 16 19:05:32.912414 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:05:32.912384    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8a563e_4735_476c_a831_c840508be692.slice/crio-eee80d151deb788b80e96d4e50fd4b567dcf0ccdec21270abc1cad9ce17e4f2d WatchSource:0}: Error finding container eee80d151deb788b80e96d4e50fd4b567dcf0ccdec21270abc1cad9ce17e4f2d: Status 404 returned error can't find the container with id eee80d151deb788b80e96d4e50fd4b567dcf0ccdec21270abc1cad9ce17e4f2d
Sep 16 19:05:33.067235 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:33.067194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb" event={"ID":"ee8a563e-4735-476c-a831-c840508be692","Type":"ContainerStarted","Data":"396717e3a731574e6c2054fab0cabde5926ed2c742956d98d802cded0f953ddd"}
Sep 16 19:05:33.067423 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:33.067257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb" event={"ID":"ee8a563e-4735-476c-a831-c840508be692","Type":"ContainerStarted","Data":"eee80d151deb788b80e96d4e50fd4b567dcf0ccdec21270abc1cad9ce17e4f2d"}
Sep 16 19:05:33.067423 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:33.067294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:05:33.088426 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:33.088369    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb" podStartSLOduration=1.088348544 podStartE2EDuration="1.088348544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:33.087728973 +0000 UTC m=+1240.534823521" watchObservedRunningTime="2026-09-16 19:05:33.088348544 +0000 UTC m=+1240.535443075"
Sep 16 19:05:37.981952 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:37.981907    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:05:47.982295 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:47.982242    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:05:57.981903 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:05:57.981848    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:06:04.074593 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:04.074561    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:06:05.555036 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:05.554986    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"]
Sep 16 19:06:07.982046 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:07.981996    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:06:14.074746 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:14.074703    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb" podUID="ee8a563e-4735-476c-a831-c840508be692" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:17.222066 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.222028    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 19:06:17.222424 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:06:17.222070    2578 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb" podUID="ee8a563e-4735-476c-a831-c840508be692" containerName="main" containerID="cri-o://396717e3a731574e6c2054fab0cabde5926ed2c742956d98d802cded0f953ddd"
Sep 16 19:06:17.222424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.222137    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb" podUID="ee8a563e-4735-476c-a831-c840508be692" containerName="main" containerID="cri-o://396717e3a731574e6c2054fab0cabde5926ed2c742956d98d802cded0f953ddd" gracePeriod=49
Sep 16 19:06:17.303570 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.303538    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb_ee8a563e-4735-476c-a831-c840508be692/main/0.log"
Sep 16 19:06:17.303746 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.303593    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee8a563e-4735-476c-a831-c840508be692" containerID="396717e3a731574e6c2054fab0cabde5926ed2c742956d98d802cded0f953ddd" exitCode=1
Sep 16 19:06:17.303746 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.303657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb" event={"ID":"ee8a563e-4735-476c-a831-c840508be692","Type":"ContainerDied","Data":"396717e3a731574e6c2054fab0cabde5926ed2c742956d98d802cded0f953ddd"}
Sep 16 19:06:17.365020 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.364997    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb_ee8a563e-4735-476c-a831-c840508be692/main/0.log"
Sep 16 19:06:17.366571 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.366554    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:06:17.480501 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.480397    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8a563e-4735-476c-a831-c840508be692-tls-certs\") pod \"ee8a563e-4735-476c-a831-c840508be692\" (UID: \"ee8a563e-4735-476c-a831-c840508be692\") "
Sep 16 19:06:17.480501 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.480465    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k69gp\" (UniqueName: \"kubernetes.io/projected/ee8a563e-4735-476c-a831-c840508be692-kube-api-access-k69gp\") pod \"ee8a563e-4735-476c-a831-c840508be692\" (UID: \"ee8a563e-4735-476c-a831-c840508be692\") "
Sep 16 19:06:17.482783 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.482747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee8a563e-4735-476c-a831-c840508be692-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee8a563e-4735-476c-a831-c840508be692" (UID: "ee8a563e-4735-476c-a831-c840508be692"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:17.482906 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.482805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8a563e-4735-476c-a831-c840508be692-kube-api-access-k69gp" (OuterVolumeSpecName: "kube-api-access-k69gp") pod "ee8a563e-4735-476c-a831-c840508be692" (UID: "ee8a563e-4735-476c-a831-c840508be692"). InnerVolumeSpecName "kube-api-access-k69gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:17.581505 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.581440    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8a563e-4735-476c-a831-c840508be692-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:06:17.581505 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.581490    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69gp\" (UniqueName: \"kubernetes.io/projected/ee8a563e-4735-476c-a831-c840508be692-kube-api-access-k69gp\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:06:17.981900 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:17.981845    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:06:18.309831 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:18.309803    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb_ee8a563e-4735-476c-a831-c840508be692/main/0.log"
Sep 16 19:06:18.310238 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:18.309907    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"
Sep 16 19:06:18.310238 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:18.309907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb" event={"ID":"ee8a563e-4735-476c-a831-c840508be692","Type":"ContainerDied","Data":"eee80d151deb788b80e96d4e50fd4b567dcf0ccdec21270abc1cad9ce17e4f2d"}
Sep 16 19:06:18.310238 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:18.310054    2578 scope.go:117] "RemoveContainer" containerID="396717e3a731574e6c2054fab0cabde5926ed2c742956d98d802cded0f953ddd"
Sep 16 19:06:18.343690 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:18.343650    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"]
Sep 16 19:06:18.357655 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:18.357621    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5dhdcb"]
Sep 16 19:06:19.064786 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:19.064743    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee8a563e-4735-476c-a831-c840508be692" path="/var/lib/kubelet/pods/ee8a563e-4735-476c-a831-c840508be692/volumes"
Sep 16 19:06:27.981786 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:27.981733    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:06:37.981809 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:37.981761    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:06:47.981492 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:47.981432    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:06:57.981663 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:06:57.981609    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 16 19:07:07.991821 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:07.991787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:07:07.999917 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:07.999885    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:07:22.192882 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:22.192804    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"]
Sep 16 19:07:37.215801 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.215754    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main" containerID="cri-o://7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e" gracePeriod=45
Sep 16 19:07:37.965668 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.965643    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:07:37.981380 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981355    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-model-cache\") pod \"be8d73b7-9bf0-486b-a339-762203c7c717\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") "
Sep 16 19:07:37.981545 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8mhf\" (UniqueName: \"kubernetes.io/projected/be8d73b7-9bf0-486b-a339-762203c7c717-kube-api-access-j8mhf\") pod \"be8d73b7-9bf0-486b-a339-762203c7c717\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") "
Sep 16 19:07:37.981545 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d73b7-9bf0-486b-a339-762203c7c717-tls-certs\") pod \"be8d73b7-9bf0-486b-a339-762203c7c717\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") "
Sep 16 19:07:37.981545 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981443    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-dshm\") pod \"be8d73b7-9bf0-486b-a339-762203c7c717\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") "
Sep 16 19:07:37.981697 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-tmp-dir\") pod \"be8d73b7-9bf0-486b-a339-762203c7c717\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") "
Sep 16 19:07:37.981697 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d73b7-9bf0-486b-a339-762203c7c717-cabundle-cert\") pod \"be8d73b7-9bf0-486b-a339-762203c7c717\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") "
Sep 16 19:07:37.981697 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-kserve-provision-location\") pod \"be8d73b7-9bf0-486b-a339-762203c7c717\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") "
Sep 16 19:07:37.981697 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981656    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-home\") pod \"be8d73b7-9bf0-486b-a339-762203c7c717\" (UID: \"be8d73b7-9bf0-486b-a339-762203c7c717\") "
Sep 16 19:07:37.981909 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981685    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-model-cache" (OuterVolumeSpecName: "model-cache") pod "be8d73b7-9bf0-486b-a339-762203c7c717" (UID: "be8d73b7-9bf0-486b-a339-762203c7c717"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:37.981963 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981938    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:07:37.982023 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.981978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8d73b7-9bf0-486b-a339-762203c7c717-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be8d73b7-9bf0-486b-a339-762203c7c717" (UID: "be8d73b7-9bf0-486b-a339-762203c7c717"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:37.982595 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.982566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-home" (OuterVolumeSpecName: "home") pod "be8d73b7-9bf0-486b-a339-762203c7c717" (UID: "be8d73b7-9bf0-486b-a339-762203c7c717"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:37.984125 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.983934    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-dshm" (OuterVolumeSpecName: "dshm") pod "be8d73b7-9bf0-486b-a339-762203c7c717" (UID: "be8d73b7-9bf0-486b-a339-762203c7c717"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:37.984125 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.983992    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8d73b7-9bf0-486b-a339-762203c7c717-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be8d73b7-9bf0-486b-a339-762203c7c717" (UID: "be8d73b7-9bf0-486b-a339-762203c7c717"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:37.984125 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.984082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8d73b7-9bf0-486b-a339-762203c7c717-kube-api-access-j8mhf" (OuterVolumeSpecName: "kube-api-access-j8mhf") pod "be8d73b7-9bf0-486b-a339-762203c7c717" (UID: "be8d73b7-9bf0-486b-a339-762203c7c717"). InnerVolumeSpecName "kube-api-access-j8mhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:37.997221 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:37.997145    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be8d73b7-9bf0-486b-a339-762203c7c717" (UID: "be8d73b7-9bf0-486b-a339-762203c7c717"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:38.013409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.013368    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be8d73b7-9bf0-486b-a339-762203c7c717" (UID: "be8d73b7-9bf0-486b-a339-762203c7c717"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:38.083047 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.083001    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8mhf\" (UniqueName: \"kubernetes.io/projected/be8d73b7-9bf0-486b-a339-762203c7c717-kube-api-access-j8mhf\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:07:38.083047 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.083032    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d73b7-9bf0-486b-a339-762203c7c717-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:07:38.083047 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.083042    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:07:38.083047 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.083050    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:07:38.083047 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.083058    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d73b7-9bf0-486b-a339-762203c7c717-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:07:38.083342 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.083066    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:07:38.083342 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.083074    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be8d73b7-9bf0-486b-a339-762203c7c717-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:07:38.720819 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.720786    2578 generic.go:356] "Generic (PLEG): container finished" podID="be8d73b7-9bf0-486b-a339-762203c7c717" containerID="7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e" exitCode=0
Sep 16 19:07:38.721240 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.720840    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"
Sep 16 19:07:38.721240 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.720853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" event={"ID":"be8d73b7-9bf0-486b-a339-762203c7c717","Type":"ContainerDied","Data":"7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e"}
Sep 16 19:07:38.721240 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.720894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q" event={"ID":"be8d73b7-9bf0-486b-a339-762203c7c717","Type":"ContainerDied","Data":"35e9b8b6353e2a523c8606f9c14064ad686ff73c2ba227530d01e9cc4cbf94f2"}
Sep 16 19:07:38.721240 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.720915    2578 scope.go:117] "RemoveContainer" containerID="7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e"
Sep 16 19:07:38.729844 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.729827    2578 scope.go:117] "RemoveContainer" containerID="ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624"
Sep 16 19:07:38.737980 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.737961    2578 scope.go:117] "RemoveContainer" containerID="7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e"
Sep 16 19:07:38.738264 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:07:38.738247    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e\": container with ID starting with 7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e not found: ID does not exist" containerID="7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e"
Sep 16 19:07:38.738310 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.738272    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e"} err="failed to get container status \"7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e\": rpc error: code = NotFound desc = could not find container \"7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e\": container with ID starting with 7109be21e859a0a9190c2a369babdcb709f26bd2039887717519cc0fb5c2727e not found: ID does not exist"
Sep 16 19:07:38.738310 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.738287    2578 scope.go:117] "RemoveContainer" containerID="ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624"
Sep 16 19:07:38.738558 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:07:38.738540    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624\": container with ID starting with ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624 not found: ID does not exist" containerID="ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624"
Sep 16 19:07:38.738621 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.738562    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624"} err="failed to get container status \"ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624\": rpc error: code = NotFound desc = could not find container \"ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624\": container with ID starting with ee56d27cfac750c10b99cb71bb5654fb1b51392b91c71f3f1a260e3d6c4fd624 not found: ID does not exist"
Sep 16 19:07:38.745000 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.744973    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"]
Sep 16 19:07:38.749260 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:38.749238    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8677dbd987-vgc5q"]
Sep 16 19:07:39.063651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:07:39.063607    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" path="/var/lib/kubelet/pods/be8d73b7-9bf0-486b-a339-762203c7c717/volumes"
Sep 16 19:08:19.376000 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.375967    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"]
Sep 16 19:08:19.376440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.376317    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="storage-initializer"
Sep 16 19:08:19.376440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.376328    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="storage-initializer"
Sep 16 19:08:19.376440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.376341    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main"
Sep 16 19:08:19.376440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.376347    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main"
Sep 16 19:08:19.376440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.376356    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8a563e-4735-476c-a831-c840508be692" containerName="main"
Sep 16 19:08:19.376440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.376362    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8a563e-4735-476c-a831-c840508be692" containerName="main"
Sep 16 19:08:19.376440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.376420    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8a563e-4735-476c-a831-c840508be692" containerName="main"
Sep 16 19:08:19.376440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.376429    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8d73b7-9bf0-486b-a339-762203c7c717" containerName="main"
Sep 16 19:08:19.379216 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.379199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.382342 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.382318    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-92jmj\""
Sep 16 19:08:19.382342 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.382333    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:08:19.382557 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.382364    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:08:19.382557 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.382333    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:08:19.382557 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.382456    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 19:08:19.382675 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.382626    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:08:19.394350 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.394326    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"]
Sep 16 19:08:19.543100 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.543051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-home\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.543282 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.543126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-kserve-provision-location\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.543282 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.543202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-model-cache\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.543377 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.543279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-dshm\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.543377 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.543306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-tmp-dir\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.543377 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.543327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996a40de-2282-4da7-9797-e5c1daba0a94-tls-certs\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.543377 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.543358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996a40de-2282-4da7-9797-e5c1daba0a94-cabundle-cert\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.543581 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.543413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrv55\" (UniqueName: \"kubernetes.io/projected/996a40de-2282-4da7-9797-e5c1daba0a94-kube-api-access-rrv55\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.644549 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.644432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-model-cache\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.644549 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.644497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-dshm\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.644787 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.644611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-tmp-dir\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.644787 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.644647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996a40de-2282-4da7-9797-e5c1daba0a94-tls-certs\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.644787 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.644689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996a40de-2282-4da7-9797-e5c1daba0a94-cabundle-cert\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.644787 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.644718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrv55\" (UniqueName: \"kubernetes.io/projected/996a40de-2282-4da7-9797-e5c1daba0a94-kube-api-access-rrv55\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.645011 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.644803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-home\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.645011 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.644855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-kserve-provision-location\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.645011 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.644925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-model-cache\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.645116 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.645013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-tmp-dir\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.645116 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.645100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-home\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.645227 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.645213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-kserve-provision-location\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.645437 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.645420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996a40de-2282-4da7-9797-e5c1daba0a94-cabundle-cert\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.646795 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.646773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-dshm\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.647221 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.647205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996a40de-2282-4da7-9797-e5c1daba0a94-tls-certs\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.656881 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.656855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrv55\" (UniqueName: \"kubernetes.io/projected/996a40de-2282-4da7-9797-e5c1daba0a94-kube-api-access-rrv55\") pod \"auth-disabled-test-kserve-65977654f6-pqtp5\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.689760 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.689734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:19.973693 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:19.973660    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"]
Sep 16 19:08:19.975926 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:08:19.975902    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996a40de_2282_4da7_9797_e5c1daba0a94.slice/crio-693dee9ea45316812cc315324541a601949941930fa67cec5f031c9c0958f4ab WatchSource:0}: Error finding container 693dee9ea45316812cc315324541a601949941930fa67cec5f031c9c0958f4ab: Status 404 returned error can't find the container with id 693dee9ea45316812cc315324541a601949941930fa67cec5f031c9c0958f4ab
Sep 16 19:08:20.922658 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:20.922620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" event={"ID":"996a40de-2282-4da7-9797-e5c1daba0a94","Type":"ContainerStarted","Data":"3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a"}
Sep 16 19:08:20.923052 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:20.922678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" event={"ID":"996a40de-2282-4da7-9797-e5c1daba0a94","Type":"ContainerStarted","Data":"693dee9ea45316812cc315324541a601949941930fa67cec5f031c9c0958f4ab"}
Sep 16 19:08:22.933263 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:22.933227    2578 generic.go:356] "Generic (PLEG): container finished" podID="996a40de-2282-4da7-9797-e5c1daba0a94" containerID="3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a" exitCode=0
Sep 16 19:08:22.933651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:22.933299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" event={"ID":"996a40de-2282-4da7-9797-e5c1daba0a94","Type":"ContainerDied","Data":"3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a"}
Sep 16 19:08:23.940502 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:23.940441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" event={"ID":"996a40de-2282-4da7-9797-e5c1daba0a94","Type":"ContainerStarted","Data":"f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6"}
Sep 16 19:08:23.964962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:23.964904    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podStartSLOduration=2.787245619 podStartE2EDuration="4.96488543s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.177639811s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:23.964877703 +0000 UTC m=+1411.411972256" watchObservedRunningTime="2026-09-16 19:08:23.96488543 +0000 UTC m=+1411.411979960"
Sep 16 19:08:24.690892 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:24.690855    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:29.690281 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:29.690239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:08:29.691961 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:29.691925    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:08:39.094898 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.094862    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"]
Sep 16 19:08:39.097592 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.097572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:08:39.101812 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.101782    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:08:39.101812 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.101806    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:08:39.102017 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.101855    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-lrjw2\""
Sep 16 19:08:39.102017 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.101783    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-q7pv7\""
Sep 16 19:08:39.102135 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.102117    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:08:39.109351 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.109326    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"]
Sep 16 19:08:39.203900 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.203872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7975253-1e42-49f3-b7dd-02e533b0f729-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx\" (UID: \"e7975253-1e42-49f3-b7dd-02e533b0f729\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:08:39.204069 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.203970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsg5k\" (UniqueName: \"kubernetes.io/projected/e7975253-1e42-49f3-b7dd-02e533b0f729-kube-api-access-dsg5k\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx\" (UID: \"e7975253-1e42-49f3-b7dd-02e533b0f729\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:08:39.305214 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.305176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7975253-1e42-49f3-b7dd-02e533b0f729-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx\" (UID: \"e7975253-1e42-49f3-b7dd-02e533b0f729\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:08:39.305391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.305254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsg5k\" (UniqueName: \"kubernetes.io/projected/e7975253-1e42-49f3-b7dd-02e533b0f729-kube-api-access-dsg5k\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx\" (UID: \"e7975253-1e42-49f3-b7dd-02e533b0f729\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:08:39.307864 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.307833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7975253-1e42-49f3-b7dd-02e533b0f729-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx\" (UID: \"e7975253-1e42-49f3-b7dd-02e533b0f729\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:08:39.314881 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.314852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsg5k\" (UniqueName: \"kubernetes.io/projected/e7975253-1e42-49f3-b7dd-02e533b0f729-kube-api-access-dsg5k\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx\" (UID: \"e7975253-1e42-49f3-b7dd-02e533b0f729\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:08:39.409218 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.409132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:08:39.680027 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.679998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"]
Sep 16 19:08:39.681881 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:08:39.681853    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7975253_1e42_49f3_b7dd_02e533b0f729.slice/crio-037e55cf2b6f7612c28f2504c3618fc2e4a0d9a7c6f5af1faf9e43e90c5ad1a9 WatchSource:0}: Error finding container 037e55cf2b6f7612c28f2504c3618fc2e4a0d9a7c6f5af1faf9e43e90c5ad1a9: Status 404 returned error can't find the container with id 037e55cf2b6f7612c28f2504c3618fc2e4a0d9a7c6f5af1faf9e43e90c5ad1a9
Sep 16 19:08:39.690963 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:39.690927    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:08:40.021914 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:40.021809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" event={"ID":"e7975253-1e42-49f3-b7dd-02e533b0f729","Type":"ContainerStarted","Data":"bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557"}
Sep 16 19:08:40.021914 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:40.021912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" event={"ID":"e7975253-1e42-49f3-b7dd-02e533b0f729","Type":"ContainerStarted","Data":"037e55cf2b6f7612c28f2504c3618fc2e4a0d9a7c6f5af1faf9e43e90c5ad1a9"}
Sep 16 19:08:40.022121 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:40.021945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:08:40.053815 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:40.053760    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" podStartSLOduration=1.053742033 podStartE2EDuration="1.053742033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:40.053139109 +0000 UTC m=+1427.500233680" watchObservedRunningTime="2026-09-16 19:08:40.053742033 +0000 UTC m=+1427.500836563"
Sep 16 19:08:49.690799 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:49.690751    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:08:59.690503 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:08:59.690432    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:09:09.690957 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:09.690906    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:09:11.029788 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:11.029755    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:09:19.411419 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:19.411377    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:19.425106 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:19.425063    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"]
Sep 16 19:09:19.690849 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:19.690748    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:09:21.029827 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:21.029788    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:29.690575 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:29.690531    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:09:31.030316 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:31.030272    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:34.447564 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:34.447498    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main" containerID="cri-o://bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557" gracePeriod=45
Sep 16 19:09:39.691046 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:39.690997    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:09:41.029653 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:41.029613    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:41.030135 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:41.029726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:09:49.691032 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:49.690981    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:09:51.030613 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:51.030565    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:53.076131 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:53.076094    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:09:53.079417 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:53.079394    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:09:53.080188 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:53.080167    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:09:53.083576 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:53.083553    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:09:59.690282 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:09:59.690235    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:10:01.030367 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:01.030324    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:10:04.704255 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:04.704226    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:10:04.878668 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:04.878623    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7975253-1e42-49f3-b7dd-02e533b0f729-tls-certs\") pod \"e7975253-1e42-49f3-b7dd-02e533b0f729\" (UID: \"e7975253-1e42-49f3-b7dd-02e533b0f729\") "
Sep 16 19:10:04.878844 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:04.878681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsg5k\" (UniqueName: \"kubernetes.io/projected/e7975253-1e42-49f3-b7dd-02e533b0f729-kube-api-access-dsg5k\") pod \"e7975253-1e42-49f3-b7dd-02e533b0f729\" (UID: \"e7975253-1e42-49f3-b7dd-02e533b0f729\") "
Sep 16 19:10:04.880882 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:04.880841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7975253-1e42-49f3-b7dd-02e533b0f729-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7975253-1e42-49f3-b7dd-02e533b0f729" (UID: "e7975253-1e42-49f3-b7dd-02e533b0f729"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:04.880882 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:04.880854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7975253-1e42-49f3-b7dd-02e533b0f729-kube-api-access-dsg5k" (OuterVolumeSpecName: "kube-api-access-dsg5k") pod "e7975253-1e42-49f3-b7dd-02e533b0f729" (UID: "e7975253-1e42-49f3-b7dd-02e533b0f729"). InnerVolumeSpecName "kube-api-access-dsg5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:04.979828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:04.979788    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7975253-1e42-49f3-b7dd-02e533b0f729-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:10:04.979828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:04.979819    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsg5k\" (UniqueName: \"kubernetes.io/projected/e7975253-1e42-49f3-b7dd-02e533b0f729-kube-api-access-dsg5k\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:10:05.467747 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:05.467713    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerID="bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557" exitCode=0
Sep 16 19:10:05.467944 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:05.467763    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"
Sep 16 19:10:05.467944 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:05.467810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" event={"ID":"e7975253-1e42-49f3-b7dd-02e533b0f729","Type":"ContainerDied","Data":"bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557"}
Sep 16 19:10:05.467944 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:05.467885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx" event={"ID":"e7975253-1e42-49f3-b7dd-02e533b0f729","Type":"ContainerDied","Data":"037e55cf2b6f7612c28f2504c3618fc2e4a0d9a7c6f5af1faf9e43e90c5ad1a9"}
Sep 16 19:10:05.467944 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:05.467923    2578 scope.go:117] "RemoveContainer" containerID="bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557"
Sep 16 19:10:05.478572 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:05.478551    2578 scope.go:117] "RemoveContainer" containerID="bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557"
Sep 16 19:10:05.478938 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:10:05.478918    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557\": container with ID starting with bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557 not found: ID does not exist" containerID="bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557"
Sep 16 19:10:05.479007 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:05.478951    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557"} err="failed to get container status \"bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557\": rpc error: code = NotFound desc = could not find container \"bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557\": container with ID starting with bc70e1c52471886489b767166532788726c91b9301c312a0b531f6b42f2d5557 not found: ID does not exist"
Sep 16 19:10:05.489278 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:05.489240    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"]
Sep 16 19:10:05.495024 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:05.494987    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-7589cjx"]
Sep 16 19:10:07.064119 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:07.064082    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" path="/var/lib/kubelet/pods/e7975253-1e42-49f3-b7dd-02e533b0f729/volumes"
Sep 16 19:10:09.690233 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:09.690186    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 19:10:19.700680 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:19.700595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:10:19.708405 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:19.708381    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:10:37.066468 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.066434    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"]
Sep 16 19:10:37.067015 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.066995    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main"
Sep 16 19:10:37.067063 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.067020    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main"
Sep 16 19:10:37.067137 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.067127    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7975253-1e42-49f3-b7dd-02e533b0f729" containerName="main"
Sep 16 19:10:37.069803 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.069782    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:10:37.074224 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.074191    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:10:37.074391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.074219    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:10:37.074391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.074192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-pnvxj\""
Sep 16 19:10:37.074391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.074228    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:10:37.074391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.074190    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-4l8g2\""
Sep 16 19:10:37.078276 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.078255    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"]
Sep 16 19:10:37.164441 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.164397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7\" (UID: \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:10:37.164666 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.164547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9h5\" (UniqueName: \"kubernetes.io/projected/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-kube-api-access-tf9h5\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7\" (UID: \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:10:37.265011 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.264973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7\" (UID: \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:10:37.265182 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.265128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9h5\" (UniqueName: \"kubernetes.io/projected/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-kube-api-access-tf9h5\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7\" (UID: \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:10:37.267488 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.267440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7\" (UID: \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:10:37.277193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.277167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9h5\" (UniqueName: \"kubernetes.io/projected/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-kube-api-access-tf9h5\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7\" (UID: \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:10:37.380791 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.380705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:10:37.656444 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:37.656416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"]
Sep 16 19:10:37.658621 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:10:37.658591    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fbc52e_6cb2_4cff_963a_e5a8d9d63f3b.slice/crio-524dfbae43ec2d1d338741f9758798788f5489120c61cfcaf1d50e247c5d89e1 WatchSource:0}: Error finding container 524dfbae43ec2d1d338741f9758798788f5489120c61cfcaf1d50e247c5d89e1: Status 404 returned error can't find the container with id 524dfbae43ec2d1d338741f9758798788f5489120c61cfcaf1d50e247c5d89e1
Sep 16 19:10:38.634422 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:38.634382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" event={"ID":"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b","Type":"ContainerStarted","Data":"e6b0a047d024c8d50a43852dfa16991928ab72ee8fd8b308bea21dce02c3f66a"}
Sep 16 19:10:38.634853 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:38.634438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" event={"ID":"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b","Type":"ContainerStarted","Data":"524dfbae43ec2d1d338741f9758798788f5489120c61cfcaf1d50e247c5d89e1"}
Sep 16 19:10:38.634853 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:38.634505    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:10:38.653737 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:38.653683    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" podStartSLOduration=1.653668547 podStartE2EDuration="1.653668547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:38.651923339 +0000 UTC m=+1546.099017901" watchObservedRunningTime="2026-09-16 19:10:38.653668547 +0000 UTC m=+1546.100763087"
Sep 16 19:10:57.752025 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:10:57.751991    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"]
Sep 16 19:11:09.641683 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:09.641647    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:11:11.835522 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:11.835456    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"]
Sep 16 19:11:12.774186 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:12.774139    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main" containerID="cri-o://f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6" gracePeriod=45
Sep 16 19:11:13.520204 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.520177    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:11:13.584076 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.583981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-kserve-provision-location\") pod \"996a40de-2282-4da7-9797-e5c1daba0a94\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") "
Sep 16 19:11:13.584076 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-dshm\") pod \"996a40de-2282-4da7-9797-e5c1daba0a94\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") "
Sep 16 19:11:13.584076 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrv55\" (UniqueName: \"kubernetes.io/projected/996a40de-2282-4da7-9797-e5c1daba0a94-kube-api-access-rrv55\") pod \"996a40de-2282-4da7-9797-e5c1daba0a94\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") "
Sep 16 19:11:13.584367 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-model-cache\") pod \"996a40de-2282-4da7-9797-e5c1daba0a94\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") "
Sep 16 19:11:13.584367 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-home\") pod \"996a40de-2282-4da7-9797-e5c1daba0a94\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") "
Sep 16 19:11:13.584367 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-tmp-dir\") pod \"996a40de-2282-4da7-9797-e5c1daba0a94\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") "
Sep 16 19:11:13.584367 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996a40de-2282-4da7-9797-e5c1daba0a94-tls-certs\") pod \"996a40de-2282-4da7-9797-e5c1daba0a94\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") "
Sep 16 19:11:13.584367 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584235    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996a40de-2282-4da7-9797-e5c1daba0a94-cabundle-cert\") pod \"996a40de-2282-4da7-9797-e5c1daba0a94\" (UID: \"996a40de-2282-4da7-9797-e5c1daba0a94\") "
Sep 16 19:11:13.584667 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584387    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-model-cache" (OuterVolumeSpecName: "model-cache") pod "996a40de-2282-4da7-9797-e5c1daba0a94" (UID: "996a40de-2282-4da7-9797-e5c1daba0a94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:13.584667 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584566    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:13.584888 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996a40de-2282-4da7-9797-e5c1daba0a94-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "996a40de-2282-4da7-9797-e5c1daba0a94" (UID: "996a40de-2282-4da7-9797-e5c1daba0a94"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:13.585020 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.584937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-home" (OuterVolumeSpecName: "home") pod "996a40de-2282-4da7-9797-e5c1daba0a94" (UID: "996a40de-2282-4da7-9797-e5c1daba0a94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:13.586645 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.586622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-dshm" (OuterVolumeSpecName: "dshm") pod "996a40de-2282-4da7-9797-e5c1daba0a94" (UID: "996a40de-2282-4da7-9797-e5c1daba0a94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:13.586645 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.586636    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996a40de-2282-4da7-9797-e5c1daba0a94-kube-api-access-rrv55" (OuterVolumeSpecName: "kube-api-access-rrv55") pod "996a40de-2282-4da7-9797-e5c1daba0a94" (UID: "996a40de-2282-4da7-9797-e5c1daba0a94"). InnerVolumeSpecName "kube-api-access-rrv55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:13.586838 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.586812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996a40de-2282-4da7-9797-e5c1daba0a94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "996a40de-2282-4da7-9797-e5c1daba0a94" (UID: "996a40de-2282-4da7-9797-e5c1daba0a94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:13.596841 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.596812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "996a40de-2282-4da7-9797-e5c1daba0a94" (UID: "996a40de-2282-4da7-9797-e5c1daba0a94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:13.614219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.614186    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "996a40de-2282-4da7-9797-e5c1daba0a94" (UID: "996a40de-2282-4da7-9797-e5c1daba0a94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:13.685410 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.685378    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:13.685410 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.685408    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrv55\" (UniqueName: \"kubernetes.io/projected/996a40de-2282-4da7-9797-e5c1daba0a94-kube-api-access-rrv55\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:13.685410 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.685418    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:13.685663 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.685448    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:13.685663 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.685460    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996a40de-2282-4da7-9797-e5c1daba0a94-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:13.685663 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.685468    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996a40de-2282-4da7-9797-e5c1daba0a94-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:13.685663 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.685500    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996a40de-2282-4da7-9797-e5c1daba0a94-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:13.808008 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.807973    2578 generic.go:356] "Generic (PLEG): container finished" podID="996a40de-2282-4da7-9797-e5c1daba0a94" containerID="f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6" exitCode=0
Sep 16 19:11:13.808185 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.808020    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"
Sep 16 19:11:13.808185 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.808062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" event={"ID":"996a40de-2282-4da7-9797-e5c1daba0a94","Type":"ContainerDied","Data":"f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6"}
Sep 16 19:11:13.808185 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.808115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5" event={"ID":"996a40de-2282-4da7-9797-e5c1daba0a94","Type":"ContainerDied","Data":"693dee9ea45316812cc315324541a601949941930fa67cec5f031c9c0958f4ab"}
Sep 16 19:11:13.808185 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.808136    2578 scope.go:117] "RemoveContainer" containerID="f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6"
Sep 16 19:11:13.818292 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.818275    2578 scope.go:117] "RemoveContainer" containerID="3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a"
Sep 16 19:11:13.826090 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.826071    2578 scope.go:117] "RemoveContainer" containerID="f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6"
Sep 16 19:11:13.826353 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:11:13.826334    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6\": container with ID starting with f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6 not found: ID does not exist" containerID="f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6"
Sep 16 19:11:13.826425 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.826367    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6"} err="failed to get container status \"f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6\": rpc error: code = NotFound desc = could not find container \"f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6\": container with ID starting with f1227b686e448f8ce85daa78454ec01c9f88832290e1936e51cc82c0a89c48a6 not found: ID does not exist"
Sep 16 19:11:13.826425 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.826396    2578 scope.go:117] "RemoveContainer" containerID="3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a"
Sep 16 19:11:13.826685 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:11:13.826667    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a\": container with ID starting with 3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a not found: ID does not exist" containerID="3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a"
Sep 16 19:11:13.826737 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.826692    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a"} err="failed to get container status \"3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a\": rpc error: code = NotFound desc = could not find container \"3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a\": container with ID starting with 3e9b538e6ed29a5e3306bf0ca387b143eed9a144433c8d633bb82c56274eb78a not found: ID does not exist"
Sep 16 19:11:13.834775 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.834722    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"]
Sep 16 19:11:13.840968 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:13.840946    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-65977654f6-pqtp5"]
Sep 16 19:11:15.063950 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:15.063914    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" path="/var/lib/kubelet/pods/996a40de-2282-4da7-9797-e5c1daba0a94/volumes"
Sep 16 19:11:19.641498 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:19.641437    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" podUID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:26.859082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:26.859033    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" podUID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" containerName="main" containerID="cri-o://e6b0a047d024c8d50a43852dfa16991928ab72ee8fd8b308bea21dce02c3f66a" gracePeriod=45
Sep 16 19:11:29.641130 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:29.641090    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" podUID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:39.641180 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:39.641139    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" podUID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:39.641589 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:39.641249    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:11:49.641170 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:49.641128    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" podUID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:57.008305 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:57.008271    2578 generic.go:356] "Generic (PLEG): container finished" podID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" containerID="e6b0a047d024c8d50a43852dfa16991928ab72ee8fd8b308bea21dce02c3f66a" exitCode=0
Sep 16 19:11:57.008696 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:57.008347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" event={"ID":"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b","Type":"ContainerDied","Data":"e6b0a047d024c8d50a43852dfa16991928ab72ee8fd8b308bea21dce02c3f66a"}
Sep 16 19:11:57.118245 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:57.118217    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:11:57.268472 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:57.268364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf9h5\" (UniqueName: \"kubernetes.io/projected/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-kube-api-access-tf9h5\") pod \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\" (UID: \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\") "
Sep 16 19:11:57.268472 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:57.268430    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-tls-certs\") pod \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\" (UID: \"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b\") "
Sep 16 19:11:57.270751 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:57.270722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" (UID: "37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:57.270866 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:57.270776    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-kube-api-access-tf9h5" (OuterVolumeSpecName: "kube-api-access-tf9h5") pod "37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" (UID: "37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b"). InnerVolumeSpecName "kube-api-access-tf9h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:57.369864 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:57.369815    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9h5\" (UniqueName: \"kubernetes.io/projected/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-kube-api-access-tf9h5\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:57.369864 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:57.369857    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:11:58.014132 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:58.014104    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"
Sep 16 19:11:58.014561 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:58.014103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7" event={"ID":"37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b","Type":"ContainerDied","Data":"524dfbae43ec2d1d338741f9758798788f5489120c61cfcaf1d50e247c5d89e1"}
Sep 16 19:11:58.014561 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:58.014235    2578 scope.go:117] "RemoveContainer" containerID="e6b0a047d024c8d50a43852dfa16991928ab72ee8fd8b308bea21dce02c3f66a"
Sep 16 19:11:58.038049 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:58.038019    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"]
Sep 16 19:11:58.042069 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:58.042042    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedwhjx7"]
Sep 16 19:11:59.063832 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:11:59.063794    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" path="/var/lib/kubelet/pods/37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b/volumes"
Sep 16 19:12:01.096518 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.096469    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"]
Sep 16 19:12:01.097121 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.097101    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" containerName="main"
Sep 16 19:12:01.097171 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.097125    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" containerName="main"
Sep 16 19:12:01.097171 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.097145    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="storage-initializer"
Sep 16 19:12:01.097171 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.097155    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="storage-initializer"
Sep 16 19:12:01.097265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.097170    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main"
Sep 16 19:12:01.097265 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.097179    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main"
Sep 16 19:12:01.097331 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.097266    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37fbc52e-6cb2-4cff-963a-e5a8d9d63f3b" containerName="main"
Sep 16 19:12:01.097331 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.097278    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="996a40de-2282-4da7-9797-e5c1daba0a94" containerName="main"
Sep 16 19:12:01.101925 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.101903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.108991 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.108962    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 19:12:01.110192 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.110167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-dwlcf\""
Sep 16 19:12:01.110361 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.110216    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 19:12:01.110361 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.110263    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 16 19:12:01.120916 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.120888    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"]
Sep 16 19:12:01.199202 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.199168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-home\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.199401 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.199232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtxs\" (UniqueName: \"kubernetes.io/projected/be89a260-8555-4319-9492-e5be5139bea2-kube-api-access-4vtxs\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.199401 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.199290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.199401 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.199328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.199401 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.199354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be89a260-8555-4319-9492-e5be5139bea2-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.199401 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.199392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-dshm\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.299981 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.299932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtxs\" (UniqueName: \"kubernetes.io/projected/be89a260-8555-4319-9492-e5be5139bea2-kube-api-access-4vtxs\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.300186 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.300018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.300186 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.300063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.300318 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.300191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be89a260-8555-4319-9492-e5be5139bea2-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.300318 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.300220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-dshm\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.300318 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.300277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-home\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.300567 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.300537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.300625 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.300561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.300625 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.300599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-home\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.302639 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.302615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-dshm\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.302976 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.302954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be89a260-8555-4319-9492-e5be5139bea2-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.309710 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.309687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtxs\" (UniqueName: \"kubernetes.io/projected/be89a260-8555-4319-9492-e5be5139bea2-kube-api-access-4vtxs\") pod \"canary-v2-kserve-7f65dbbccc-b8xqd\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.418849 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.418758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:01.692418 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.692339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"]
Sep 16 19:12:01.695826 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:12:01.695796    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe89a260_8555_4319_9492_e5be5139bea2.slice/crio-a0270ba2dcc7cbe53a96ae29fc42298c22e64fd0926a2844260526480d3c7866 WatchSource:0}: Error finding container a0270ba2dcc7cbe53a96ae29fc42298c22e64fd0926a2844260526480d3c7866: Status 404 returned error can't find the container with id a0270ba2dcc7cbe53a96ae29fc42298c22e64fd0926a2844260526480d3c7866
Sep 16 19:12:01.697925 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:01.697908    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:12:02.037690 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:02.037649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd" event={"ID":"be89a260-8555-4319-9492-e5be5139bea2","Type":"ContainerStarted","Data":"a0270ba2dcc7cbe53a96ae29fc42298c22e64fd0926a2844260526480d3c7866"}
Sep 16 19:12:03.044556 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:03.044522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd" event={"ID":"be89a260-8555-4319-9492-e5be5139bea2","Type":"ContainerStarted","Data":"efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642"}
Sep 16 19:12:03.064719 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:03.064669    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd" podStartSLOduration=1.109006316 podStartE2EDuration="2.06465358s" totalImagesPullingTime="955.647264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:03.064128107 +0000 UTC m=+1630.511222639" watchObservedRunningTime="2026-09-16 19:12:03.06465358 +0000 UTC m=+1630.511748109"
Sep 16 19:12:03.419631 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:03.419424    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:11.419808 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:11.419765    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:11.432226 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:11.432198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:12.100456 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:12.100420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:12:28.544612 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.544574    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"]
Sep 16 19:12:28.547953 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.547935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.552755 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.552726    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-c2vgg\""
Sep 16 19:12:28.553418 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.553393    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:28.553686 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.553669    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:12:28.554051 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.554035    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:28.590854 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.590819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"]
Sep 16 19:12:28.636077 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.636041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.636257 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.636095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d200dd4a-9185-41b9-a2bc-95c816aac740-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.636257 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.636146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.636257 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.636204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d200dd4a-9185-41b9-a2bc-95c816aac740-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.737258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.737218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d200dd4a-9185-41b9-a2bc-95c816aac740-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.737258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.737257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.737510 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.737297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d200dd4a-9185-41b9-a2bc-95c816aac740-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.737510 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.737337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.737790 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.737768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.737837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.737804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.738020 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.737998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d200dd4a-9185-41b9-a2bc-95c816aac740-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.739963 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.739943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d200dd4a-9185-41b9-a2bc-95c816aac740-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:28.860418 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:28.860330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:12:29.144375 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:29.144354    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"]
Sep 16 19:12:29.147078 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:12:29.147052    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd200dd4a_9185_41b9_a2bc_95c816aac740.slice/crio-e0214f40b57505cd60d448572b7c4b7b363da88fb89c7d1bb5fb925dea270e05 WatchSource:0}: Error finding container e0214f40b57505cd60d448572b7c4b7b363da88fb89c7d1bb5fb925dea270e05: Status 404 returned error can't find the container with id e0214f40b57505cd60d448572b7c4b7b363da88fb89c7d1bb5fb925dea270e05
Sep 16 19:12:29.175957 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:29.175929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c" event={"ID":"d200dd4a-9185-41b9-a2bc-95c816aac740","Type":"ContainerStarted","Data":"e0214f40b57505cd60d448572b7c4b7b363da88fb89c7d1bb5fb925dea270e05"}
Sep 16 19:12:30.182131 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:30.182093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c" event={"ID":"d200dd4a-9185-41b9-a2bc-95c816aac740","Type":"ContainerStarted","Data":"450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d"}
Sep 16 19:12:32.194805 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:32.194770    2578 generic.go:356] "Generic (PLEG): container finished" podID="d200dd4a-9185-41b9-a2bc-95c816aac740" containerID="450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d" exitCode=0
Sep 16 19:12:32.195298 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:12:32.194821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c" event={"ID":"d200dd4a-9185-41b9-a2bc-95c816aac740","Type":"ContainerDied","Data":"450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d"}
Sep 16 19:13:27.521033 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:13:27.520920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c" event={"ID":"d200dd4a-9185-41b9-a2bc-95c816aac740","Type":"ContainerStarted","Data":"70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4"}
Sep 16 19:13:27.522023 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:13:27.521982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:13:27.545522 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:13:27.545461    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c" podStartSLOduration=2.486261498 podStartE2EDuration="59.545442659s" totalImagesPullingTime="54.846617715s" totalInitContainerRuntime="2.212563446s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:27.54232214 +0000 UTC m=+1714.989416671" watchObservedRunningTime="2026-09-16 19:13:27.545442659 +0000 UTC m=+1714.992537188"
Sep 16 19:13:55.159854 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:13:55.159823    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"]
Sep 16 19:13:58.538374 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:13:58.538331    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:14:10.178535 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.178491    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd" podUID="be89a260-8555-4319-9492-e5be5139bea2" containerName="main" containerID="cri-o://efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642" gracePeriod=45
Sep 16 19:14:10.436696 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.436628    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:14:10.577991 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.577945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be89a260-8555-4319-9492-e5be5139bea2-tls-certs\") pod \"be89a260-8555-4319-9492-e5be5139bea2\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") "
Sep 16 19:14:10.578189 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.578016    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vtxs\" (UniqueName: \"kubernetes.io/projected/be89a260-8555-4319-9492-e5be5139bea2-kube-api-access-4vtxs\") pod \"be89a260-8555-4319-9492-e5be5139bea2\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") "
Sep 16 19:14:10.578189 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.578079    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-model-cache\") pod \"be89a260-8555-4319-9492-e5be5139bea2\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") "
Sep 16 19:14:10.578189 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.578119    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-home\") pod \"be89a260-8555-4319-9492-e5be5139bea2\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") "
Sep 16 19:14:10.578374 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.578194    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-dshm\") pod \"be89a260-8555-4319-9492-e5be5139bea2\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") "
Sep 16 19:14:10.578374 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.578229    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-tmp-dir\") pod \"be89a260-8555-4319-9492-e5be5139bea2\" (UID: \"be89a260-8555-4319-9492-e5be5139bea2\") "
Sep 16 19:14:10.578374 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.578350    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-model-cache" (OuterVolumeSpecName: "model-cache") pod "be89a260-8555-4319-9492-e5be5139bea2" (UID: "be89a260-8555-4319-9492-e5be5139bea2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:10.578563 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.578452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-home" (OuterVolumeSpecName: "home") pod "be89a260-8555-4319-9492-e5be5139bea2" (UID: "be89a260-8555-4319-9492-e5be5139bea2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:10.578563 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.578470    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:10.578668 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.578612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be89a260-8555-4319-9492-e5be5139bea2" (UID: "be89a260-8555-4319-9492-e5be5139bea2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:10.580364 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.580334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-dshm" (OuterVolumeSpecName: "dshm") pod "be89a260-8555-4319-9492-e5be5139bea2" (UID: "be89a260-8555-4319-9492-e5be5139bea2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:10.580505 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.580390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be89a260-8555-4319-9492-e5be5139bea2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be89a260-8555-4319-9492-e5be5139bea2" (UID: "be89a260-8555-4319-9492-e5be5139bea2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:10.580505 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.580424    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be89a260-8555-4319-9492-e5be5139bea2-kube-api-access-4vtxs" (OuterVolumeSpecName: "kube-api-access-4vtxs") pod "be89a260-8555-4319-9492-e5be5139bea2" (UID: "be89a260-8555-4319-9492-e5be5139bea2"). InnerVolumeSpecName "kube-api-access-4vtxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:10.679889 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.679847    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:10.679889 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.679878    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:10.679889 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.679888    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be89a260-8555-4319-9492-e5be5139bea2-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:10.679889 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.679897    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vtxs\" (UniqueName: \"kubernetes.io/projected/be89a260-8555-4319-9492-e5be5139bea2-kube-api-access-4vtxs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:10.680159 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.679908    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be89a260-8555-4319-9492-e5be5139bea2-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:10.757014 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.756920    2578 generic.go:356] "Generic (PLEG): container finished" podID="be89a260-8555-4319-9492-e5be5139bea2" containerID="efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642" exitCode=0
Sep 16 19:14:10.757014 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.756965    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"
Sep 16 19:14:10.757014 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.757006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd" event={"ID":"be89a260-8555-4319-9492-e5be5139bea2","Type":"ContainerDied","Data":"efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642"}
Sep 16 19:14:10.757247 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.757053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd" event={"ID":"be89a260-8555-4319-9492-e5be5139bea2","Type":"ContainerDied","Data":"a0270ba2dcc7cbe53a96ae29fc42298c22e64fd0926a2844260526480d3c7866"}
Sep 16 19:14:10.757247 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.757075    2578 scope.go:117] "RemoveContainer" containerID="efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642"
Sep 16 19:14:10.765993 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.765975    2578 scope.go:117] "RemoveContainer" containerID="efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642"
Sep 16 19:14:10.766270 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:14:10.766247    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642\": container with ID starting with efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642 not found: ID does not exist" containerID="efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642"
Sep 16 19:14:10.766354 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.766284    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642"} err="failed to get container status \"efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642\": rpc error: code = NotFound desc = could not find container \"efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642\": container with ID starting with efb1f36d8d12ff038a4257322cc84d5750a2f64f75024822f38ade150a364642 not found: ID does not exist"
Sep 16 19:14:10.782722 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.782696    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"]
Sep 16 19:14:10.786114 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:10.786086    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-b8xqd"]
Sep 16 19:14:11.064315 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:11.064279    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be89a260-8555-4319-9492-e5be5139bea2" path="/var/lib/kubelet/pods/be89a260-8555-4319-9492-e5be5139bea2/volumes"
Sep 16 19:14:23.390844 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:23.390785    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c_d200dd4a-9185-41b9-a2bc-95c816aac740/main/0.log"
Sep 16 19:14:45.839127 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:45.839037    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"]
Sep 16 19:14:45.839870 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:45.839316    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c" podUID="d200dd4a-9185-41b9-a2bc-95c816aac740" containerName="main" containerID="cri-o://70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4" gracePeriod=30
Sep 16 19:14:46.407501 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.407460    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:14:46.464799 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.464766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d200dd4a-9185-41b9-a2bc-95c816aac740-tls-certs\") pod \"d200dd4a-9185-41b9-a2bc-95c816aac740\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") "
Sep 16 19:14:46.464799 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.464815    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d200dd4a-9185-41b9-a2bc-95c816aac740-cabundle-cert\") pod \"d200dd4a-9185-41b9-a2bc-95c816aac740\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") "
Sep 16 19:14:46.465066 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.465023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-tmp-dir\") pod \"d200dd4a-9185-41b9-a2bc-95c816aac740\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") "
Sep 16 19:14:46.465134 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.465081    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-kserve-provision-location\") pod \"d200dd4a-9185-41b9-a2bc-95c816aac740\" (UID: \"d200dd4a-9185-41b9-a2bc-95c816aac740\") "
Sep 16 19:14:46.465189 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.465153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d200dd4a-9185-41b9-a2bc-95c816aac740-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d200dd4a-9185-41b9-a2bc-95c816aac740" (UID: "d200dd4a-9185-41b9-a2bc-95c816aac740"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:46.465357 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.465339    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d200dd4a-9185-41b9-a2bc-95c816aac740-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.465430 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.465403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d200dd4a-9185-41b9-a2bc-95c816aac740" (UID: "d200dd4a-9185-41b9-a2bc-95c816aac740"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.466984 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.466953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d200dd4a-9185-41b9-a2bc-95c816aac740-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d200dd4a-9185-41b9-a2bc-95c816aac740" (UID: "d200dd4a-9185-41b9-a2bc-95c816aac740"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:46.490288 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.490187    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d200dd4a-9185-41b9-a2bc-95c816aac740" (UID: "d200dd4a-9185-41b9-a2bc-95c816aac740"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.566008 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.565970    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.566008 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.566005    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d200dd4a-9185-41b9-a2bc-95c816aac740-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.566008 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.566014    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d200dd4a-9185-41b9-a2bc-95c816aac740-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.936703 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.936668    2578 generic.go:356] "Generic (PLEG): container finished" podID="d200dd4a-9185-41b9-a2bc-95c816aac740" containerID="70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4" exitCode=0
Sep 16 19:14:46.937095 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.936717    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"
Sep 16 19:14:46.937095 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.936749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c" event={"ID":"d200dd4a-9185-41b9-a2bc-95c816aac740","Type":"ContainerDied","Data":"70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4"}
Sep 16 19:14:46.937095 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.936803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c" event={"ID":"d200dd4a-9185-41b9-a2bc-95c816aac740","Type":"ContainerDied","Data":"e0214f40b57505cd60d448572b7c4b7b363da88fb89c7d1bb5fb925dea270e05"}
Sep 16 19:14:46.937095 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.936826    2578 scope.go:117] "RemoveContainer" containerID="70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4"
Sep 16 19:14:46.945903 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.945886    2578 scope.go:117] "RemoveContainer" containerID="450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d"
Sep 16 19:14:46.953316 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.953299    2578 scope.go:117] "RemoveContainer" containerID="70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4"
Sep 16 19:14:46.953582 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:14:46.953564    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4\": container with ID starting with 70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4 not found: ID does not exist" containerID="70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4"
Sep 16 19:14:46.953638 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.953591    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4"} err="failed to get container status \"70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4\": rpc error: code = NotFound desc = could not find container \"70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4\": container with ID starting with 70dca1bf102739ee27eca19efc4cf3454a013994c3c79b368cd946926aab03b4 not found: ID does not exist"
Sep 16 19:14:46.953638 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.953606    2578 scope.go:117] "RemoveContainer" containerID="450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d"
Sep 16 19:14:46.953833 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:14:46.953814    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d\": container with ID starting with 450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d not found: ID does not exist" containerID="450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d"
Sep 16 19:14:46.953887 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.953843    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d"} err="failed to get container status \"450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d\": rpc error: code = NotFound desc = could not find container \"450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d\": container with ID starting with 450b34c0ad0f5219aaca03f8b1b78884b66809be613ece229348bacbc600bf5d not found: ID does not exist"
Sep 16 19:14:46.958695 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.958673    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"]
Sep 16 19:14:46.965392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:46.965366    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54f587dc94-bld4c"]
Sep 16 19:14:47.064022 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:47.063989    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d200dd4a-9185-41b9-a2bc-95c816aac740" path="/var/lib/kubelet/pods/d200dd4a-9185-41b9-a2bc-95c816aac740/volumes"
Sep 16 19:14:53.106084 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:53.106057    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:14:53.110038 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:53.110014    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:14:53.112667 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:53.112644    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:14:53.116942 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:14:53.116923    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:15:00.447295 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.447255    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"]
Sep 16 19:15:00.447702 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.447657    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d200dd4a-9185-41b9-a2bc-95c816aac740" containerName="main"
Sep 16 19:15:00.447702 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.447672    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d200dd4a-9185-41b9-a2bc-95c816aac740" containerName="main"
Sep 16 19:15:00.447702 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.447681    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be89a260-8555-4319-9492-e5be5139bea2" containerName="main"
Sep 16 19:15:00.447702 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.447686    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="be89a260-8555-4319-9492-e5be5139bea2" containerName="main"
Sep 16 19:15:00.447838 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.447704    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d200dd4a-9185-41b9-a2bc-95c816aac740" containerName="storage-initializer"
Sep 16 19:15:00.447838 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.447711    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d200dd4a-9185-41b9-a2bc-95c816aac740" containerName="storage-initializer"
Sep 16 19:15:00.447838 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.447763    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="be89a260-8555-4319-9492-e5be5139bea2" containerName="main"
Sep 16 19:15:00.447838 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.447769    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d200dd4a-9185-41b9-a2bc-95c816aac740" containerName="main"
Sep 16 19:15:00.451174 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.451151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.454212 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.454188    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 19:15:00.454324 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.454267    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 19:15:00.455512 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.455487    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 19:15:00.455620 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.455577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-wc99b\""
Sep 16 19:15:00.465713 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.465687    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"]
Sep 16 19:15:00.594091 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.594051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-dshm\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.594263 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.594098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-model-cache\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.594263 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.594133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tls-certs\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.594263 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.594149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nxp7\" (UniqueName: \"kubernetes.io/projected/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-kube-api-access-5nxp7\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.594263 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.594176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-home\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.594263 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.594226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.695626 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.695583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-home\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.695822 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.695641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.695822 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.695682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-dshm\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.695822 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.695707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-model-cache\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.695822 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.695726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tls-certs\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.696000 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.695823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nxp7\" (UniqueName: \"kubernetes.io/projected/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-kube-api-access-5nxp7\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.696057 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.696028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-home\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.696161 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.696138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-model-cache\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.696227 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.696185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.698128 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.698069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-dshm\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.698323 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.698307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tls-certs\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.704355 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.704326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nxp7\" (UniqueName: \"kubernetes.io/projected/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-kube-api-access-5nxp7\") pod \"test-div-v1-kserve-fb566d667-2bbvz\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:00.762165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:00.762128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:01.048681 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:01.048654    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"]
Sep 16 19:15:01.053277 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:15:01.053240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c0d8ce_8e8a_40b1_9a37_5dc94d402bc2.slice/crio-8bdadfa7a80a004c46fb337e7471d45ba00348854609d9216093214561300853 WatchSource:0}: Error finding container 8bdadfa7a80a004c46fb337e7471d45ba00348854609d9216093214561300853: Status 404 returned error can't find the container with id 8bdadfa7a80a004c46fb337e7471d45ba00348854609d9216093214561300853
Sep 16 19:15:02.011981 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:02.011945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz" event={"ID":"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2","Type":"ContainerStarted","Data":"dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7"}
Sep 16 19:15:02.012396 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:02.012032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz" event={"ID":"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2","Type":"ContainerStarted","Data":"8bdadfa7a80a004c46fb337e7471d45ba00348854609d9216093214561300853"}
Sep 16 19:15:02.038631 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:02.038570    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz" podStartSLOduration=2.038554777 podStartE2EDuration="2.038554777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:02.036110942 +0000 UTC m=+1809.483205473" watchObservedRunningTime="2026-09-16 19:15:02.038554777 +0000 UTC m=+1809.485649307"
Sep 16 19:15:02.762466 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:02.762412    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:10.763338 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:10.763291    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:10.776112 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:10.776081    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:15:11.067101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:15:11.067072    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:16:02.283725 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.283691    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"]
Sep 16 19:16:02.286341 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.286321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.289241 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.289219    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:16:02.289531 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.289512    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:16:02.290596 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.290576    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-hqbzm\""
Sep 16 19:16:02.290724 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.290638    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:16:02.290724 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.290657    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:16:02.290724 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.290643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:16:02.301805 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.301772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"]
Sep 16 19:16:02.444235 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.444190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-home\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.444235 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.444232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-model-cache\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.444461 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.444307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c64343-7937-4070-bdff-24b222026360-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.444461 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.444346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96ng\" (UniqueName: \"kubernetes.io/projected/08c64343-7937-4070-bdff-24b222026360-kube-api-access-f96ng\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.444461 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.444371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08c64343-7937-4070-bdff-24b222026360-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.444461 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.444418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-dshm\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.444461 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.444439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.444461 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.444459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.545698 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.545604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c64343-7937-4070-bdff-24b222026360-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.545698 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.545646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96ng\" (UniqueName: \"kubernetes.io/projected/08c64343-7937-4070-bdff-24b222026360-kube-api-access-f96ng\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.545908 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.545767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08c64343-7937-4070-bdff-24b222026360-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.545908 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.545823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-dshm\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.545908 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.545846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.545908 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.545869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.546384 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.545926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-home\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.546384 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.545967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-model-cache\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.546384 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.546345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.546633 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.546422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-model-cache\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.546633 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.546592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-home\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.546633 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.546616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.546990 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.546940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08c64343-7937-4070-bdff-24b222026360-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.548646 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.548622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-dshm\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.549178 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.549156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c64343-7937-4070-bdff-24b222026360-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.560813 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.560748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96ng\" (UniqueName: \"kubernetes.io/projected/08c64343-7937-4070-bdff-24b222026360-kube-api-access-f96ng\") pod \"tokenizer-migration-path-test-kserve-6cdb585d6-rh99x\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.562345 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.562316    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"]
Sep 16 19:16:02.565396 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.565371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.580443 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.580418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"]
Sep 16 19:16:02.597609 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.597574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:02.747819 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.747760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.748015 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.747835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.748015 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.747894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.748133 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.748004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.848939 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.848841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.848939 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.848893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.848939 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.848927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.849234 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.848969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.849369 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.849337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.849434 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.849352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.849723 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.849706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.851509 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.851448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:02.878828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.878801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"]
Sep 16 19:16:02.880831 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:16:02.880795    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c64343_7937_4070_bdff_24b222026360.slice/crio-b667502778420b4157e8f11ab35eb6a33d7a1f2268a6cd2beae5ad1b0f8a0d2e WatchSource:0}: Error finding container b667502778420b4157e8f11ab35eb6a33d7a1f2268a6cd2beae5ad1b0f8a0d2e: Status 404 returned error can't find the container with id b667502778420b4157e8f11ab35eb6a33d7a1f2268a6cd2beae5ad1b0f8a0d2e
Sep 16 19:16:02.884577 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:02.884559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:03.165413 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:03.165379    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"]
Sep 16 19:16:03.166277 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:16:03.166252    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2376ac4a_f5bc_4068_a921_44c8fa3f9bc5.slice/crio-08cf4f6c368cf38b1694b97a00fab85c315b6351cfad1ab7aa7cd4ad54f25602 WatchSource:0}: Error finding container 08cf4f6c368cf38b1694b97a00fab85c315b6351cfad1ab7aa7cd4ad54f25602: Status 404 returned error can't find the container with id 08cf4f6c368cf38b1694b97a00fab85c315b6351cfad1ab7aa7cd4ad54f25602
Sep 16 19:16:03.309202 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:03.309165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" event={"ID":"08c64343-7937-4070-bdff-24b222026360","Type":"ContainerStarted","Data":"b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21"}
Sep 16 19:16:03.309697 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:03.309230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" event={"ID":"08c64343-7937-4070-bdff-24b222026360","Type":"ContainerStarted","Data":"b667502778420b4157e8f11ab35eb6a33d7a1f2268a6cd2beae5ad1b0f8a0d2e"}
Sep 16 19:16:03.311165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:03.311125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh" event={"ID":"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5","Type":"ContainerStarted","Data":"acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a"}
Sep 16 19:16:03.311294 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:03.311178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh" event={"ID":"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5","Type":"ContainerStarted","Data":"08cf4f6c368cf38b1694b97a00fab85c315b6351cfad1ab7aa7cd4ad54f25602"}
Sep 16 19:16:06.329396 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:06.329357    2578 generic.go:356] "Generic (PLEG): container finished" podID="2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" containerID="acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a" exitCode=0
Sep 16 19:16:06.329866 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:06.329441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh" event={"ID":"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5","Type":"ContainerDied","Data":"acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a"}
Sep 16 19:16:06.331196 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:06.331173    2578 generic.go:356] "Generic (PLEG): container finished" podID="08c64343-7937-4070-bdff-24b222026360" containerID="b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21" exitCode=0
Sep 16 19:16:06.331296 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:06.331243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" event={"ID":"08c64343-7937-4070-bdff-24b222026360","Type":"ContainerDied","Data":"b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21"}
Sep 16 19:16:07.338620 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:07.338584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh" event={"ID":"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5","Type":"ContainerStarted","Data":"7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61"}
Sep 16 19:16:07.339220 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:07.338823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:07.340730 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:07.340708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" event={"ID":"08c64343-7937-4070-bdff-24b222026360","Type":"ContainerStarted","Data":"cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a"}
Sep 16 19:16:07.360922 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:07.360869    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh" podStartSLOduration=2.851018277 podStartE2EDuration="5.360853222s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.509834945s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:07.359793661 +0000 UTC m=+1874.806888212" watchObservedRunningTime="2026-09-16 19:16:07.360853222 +0000 UTC m=+1874.807947751"
Sep 16 19:16:07.598038 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:07.597934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:12.598268 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:12.598202    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:12.611149 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:12.611109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:12.646382 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:12.646323    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" podStartSLOduration=8.096392486 podStartE2EDuration="10.646300411s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.549907925s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:07.38406145 +0000 UTC m=+1874.831155981" watchObservedRunningTime="2026-09-16 19:16:12.646300411 +0000 UTC m=+1880.093394957"
Sep 16 19:16:13.386522 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:13.386474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:16:24.414563 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:24.414472    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"]
Sep 16 19:16:38.356994 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:38.356959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:16:39.434370 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.434325    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz" podUID="14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" containerName="main" containerID="cri-o://dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7" gracePeriod=45
Sep 16 19:16:39.681462 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.681438    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:16:39.787384 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nxp7\" (UniqueName: \"kubernetes.io/projected/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-kube-api-access-5nxp7\") pod \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") "
Sep 16 19:16:39.787612 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787398    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tls-certs\") pod \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") "
Sep 16 19:16:39.787612 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-home\") pod \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") "
Sep 16 19:16:39.787612 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787568    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tmp-dir\") pod \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") "
Sep 16 19:16:39.787612 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787606    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-dshm\") pod \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") "
Sep 16 19:16:39.787840 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787622    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-model-cache\") pod \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\" (UID: \"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2\") "
Sep 16 19:16:39.787840 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787803    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-home" (OuterVolumeSpecName: "home") pod "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" (UID: "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:39.787953 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" (UID: "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:39.787953 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787891    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.787953 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787928    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.788111 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.787953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" (UID: "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:39.789865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.789827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" (UID: "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:39.789979 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.789865    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-kube-api-access-5nxp7" (OuterVolumeSpecName: "kube-api-access-5nxp7") pod "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" (UID: "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2"). InnerVolumeSpecName "kube-api-access-5nxp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:39.789979 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.789908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-dshm" (OuterVolumeSpecName: "dshm") pod "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" (UID: "14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:39.888333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.888294    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.888333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.888325    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.888333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.888339    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nxp7\" (UniqueName: \"kubernetes.io/projected/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-kube-api-access-5nxp7\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:16:39.888614 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:39.888352    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:16:40.512023 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:40.511990    2578 generic.go:356] "Generic (PLEG): container finished" podID="14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" containerID="dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7" exitCode=0
Sep 16 19:16:40.512551 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:40.512039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"
Sep 16 19:16:40.512551 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:40.512080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz" event={"ID":"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2","Type":"ContainerDied","Data":"dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7"}
Sep 16 19:16:40.512551 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:40.512140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz" event={"ID":"14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2","Type":"ContainerDied","Data":"8bdadfa7a80a004c46fb337e7471d45ba00348854609d9216093214561300853"}
Sep 16 19:16:40.512551 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:40.512173    2578 scope.go:117] "RemoveContainer" containerID="dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7"
Sep 16 19:16:40.521210 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:40.521184    2578 scope.go:117] "RemoveContainer" containerID="dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7"
Sep 16 19:16:40.521507 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:16:40.521473    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7\": container with ID starting with dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7 not found: ID does not exist" containerID="dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7"
Sep 16 19:16:40.521565 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:40.521516    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7"} err="failed to get container status \"dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7\": rpc error: code = NotFound desc = could not find container \"dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7\": container with ID starting with dc4eb168a0ec9293ed1f248ee12948af0553d0eaf68b3ca802e12a130b391be7 not found: ID does not exist"
Sep 16 19:16:40.534942 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:40.534915    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"]
Sep 16 19:16:40.538896 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:40.538874    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2bbvz"]
Sep 16 19:16:41.065765 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:16:41.065728    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" path="/var/lib/kubelet/pods/14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2/volumes"
Sep 16 19:17:14.848144 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:14.848111    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh_2376ac4a-f5bc-4068-a921-44c8fa3f9bc5/main/0.log"
Sep 16 19:17:31.389646 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.389615    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"]
Sep 16 19:17:31.399411 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.399381    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"]
Sep 16 19:17:31.399710 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.399670    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh" podUID="2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" containerName="main" containerID="cri-o://7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61" gracePeriod=30
Sep 16 19:17:31.400776 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:17:31.400747    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:17:31.400776 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:31.400776 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:31.400776 ip-10-0-143-161 kubenswrapper[2578]:  > containerID="cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a" cmd=["/bin/sleep","15"]
Sep 16 19:17:31.401031 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.400794    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" message="" err=<
Sep 16 19:17:31.401031 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:31.401031 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:31.401031 ip-10-0-143-161 kubenswrapper[2578]:  >
Sep 16 19:17:31.401031 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:17:31.400807    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:17:31.401031 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:31.401031 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:31.401031 ip-10-0-143-161 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" podUID="08c64343-7937-4070-bdff-24b222026360" containerName="main" containerID="cri-o://cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a"
Sep 16 19:17:31.401031 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.400841    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" podUID="08c64343-7937-4070-bdff-24b222026360" containerName="main" containerID="cri-o://cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a" gracePeriod=60
Sep 16 19:17:31.754624 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.754598    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:17:31.774060 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.774019    2578 generic.go:356] "Generic (PLEG): container finished" podID="08c64343-7937-4070-bdff-24b222026360" containerID="cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a" exitCode=0
Sep 16 19:17:31.774208 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.774075    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"
Sep 16 19:17:31.774208 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.774112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" event={"ID":"08c64343-7937-4070-bdff-24b222026360","Type":"ContainerDied","Data":"cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a"}
Sep 16 19:17:31.774208 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.774179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x" event={"ID":"08c64343-7937-4070-bdff-24b222026360","Type":"ContainerDied","Data":"b667502778420b4157e8f11ab35eb6a33d7a1f2268a6cd2beae5ad1b0f8a0d2e"}
Sep 16 19:17:31.774311 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.774211    2578 scope.go:117] "RemoveContainer" containerID="cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a"
Sep 16 19:17:31.784096 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.784068    2578 scope.go:117] "RemoveContainer" containerID="b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21"
Sep 16 19:17:31.793421 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.793396    2578 scope.go:117] "RemoveContainer" containerID="cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a"
Sep 16 19:17:31.793948 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:17:31.793925    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a\": container with ID starting with cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a not found: ID does not exist" containerID="cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a"
Sep 16 19:17:31.794045 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.793957    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a"} err="failed to get container status \"cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a\": rpc error: code = NotFound desc = could not find container \"cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a\": container with ID starting with cf4fc8a95c71ca48c56fa905f169effd823c0450c4c032b3922adcf2dfc3457a not found: ID does not exist"
Sep 16 19:17:31.794045 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.793977    2578 scope.go:117] "RemoveContainer" containerID="b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21"
Sep 16 19:17:31.794398 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:17:31.794361    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21\": container with ID starting with b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21 not found: ID does not exist" containerID="b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21"
Sep 16 19:17:31.794504 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.794406    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21"} err="failed to get container status \"b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21\": rpc error: code = NotFound desc = could not find container \"b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21\": container with ID starting with b95666bbafe51d44ed24ab5b3098746597b448b39e3206fca753185c326e6c21 not found: ID does not exist"
Sep 16 19:17:31.851030 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-kserve-provision-location\") pod \"08c64343-7937-4070-bdff-24b222026360\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") "
Sep 16 19:17:31.851128 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c64343-7937-4070-bdff-24b222026360-tls-certs\") pod \"08c64343-7937-4070-bdff-24b222026360\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") "
Sep 16 19:17:31.851128 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-home\") pod \"08c64343-7937-4070-bdff-24b222026360\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") "
Sep 16 19:17:31.851128 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f96ng\" (UniqueName: \"kubernetes.io/projected/08c64343-7937-4070-bdff-24b222026360-kube-api-access-f96ng\") pod \"08c64343-7937-4070-bdff-24b222026360\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") "
Sep 16 19:17:31.851289 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-tmp-dir\") pod \"08c64343-7937-4070-bdff-24b222026360\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") "
Sep 16 19:17:31.851289 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08c64343-7937-4070-bdff-24b222026360-cabundle-cert\") pod \"08c64343-7937-4070-bdff-24b222026360\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") "
Sep 16 19:17:31.851399 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-dshm\") pod \"08c64343-7937-4070-bdff-24b222026360\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") "
Sep 16 19:17:31.851399 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-model-cache\") pod \"08c64343-7937-4070-bdff-24b222026360\" (UID: \"08c64343-7937-4070-bdff-24b222026360\") "
Sep 16 19:17:31.851599 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851423    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08c64343-7937-4070-bdff-24b222026360" (UID: "08c64343-7937-4070-bdff-24b222026360"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.851793 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851709    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-model-cache" (OuterVolumeSpecName: "model-cache") pod "08c64343-7937-4070-bdff-24b222026360" (UID: "08c64343-7937-4070-bdff-24b222026360"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.852006 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.851984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c64343-7937-4070-bdff-24b222026360-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08c64343-7937-4070-bdff-24b222026360" (UID: "08c64343-7937-4070-bdff-24b222026360"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:31.852118 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.852086    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-home" (OuterVolumeSpecName: "home") pod "08c64343-7937-4070-bdff-24b222026360" (UID: "08c64343-7937-4070-bdff-24b222026360"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.852193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.852099    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.852193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.852143    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.852193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.852160    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08c64343-7937-4070-bdff-24b222026360-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.853717 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.853690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c64343-7937-4070-bdff-24b222026360-kube-api-access-f96ng" (OuterVolumeSpecName: "kube-api-access-f96ng") pod "08c64343-7937-4070-bdff-24b222026360" (UID: "08c64343-7937-4070-bdff-24b222026360"). InnerVolumeSpecName "kube-api-access-f96ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:31.853717 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.853701    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-dshm" (OuterVolumeSpecName: "dshm") pod "08c64343-7937-4070-bdff-24b222026360" (UID: "08c64343-7937-4070-bdff-24b222026360"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.854052 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.854035    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c64343-7937-4070-bdff-24b222026360-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08c64343-7937-4070-bdff-24b222026360" (UID: "08c64343-7937-4070-bdff-24b222026360"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:31.875180 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.875137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08c64343-7937-4070-bdff-24b222026360" (UID: "08c64343-7937-4070-bdff-24b222026360"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:31.953378 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.953294    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.953378 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.953325    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.953378 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.953336    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c64343-7937-4070-bdff-24b222026360-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.953378 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.953345    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c64343-7937-4070-bdff-24b222026360-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.953378 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.953353    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96ng\" (UniqueName: \"kubernetes.io/projected/08c64343-7937-4070-bdff-24b222026360-kube-api-access-f96ng\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:31.975994 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:31.975968    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:17:32.054072 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.054034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tmp-dir\") pod \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") "
Sep 16 19:17:32.054072 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.054073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-kserve-provision-location\") pod \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") "
Sep 16 19:17:32.054291 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.054129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-cabundle-cert\") pod \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") "
Sep 16 19:17:32.054291 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.054152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tls-certs\") pod \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\" (UID: \"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5\") "
Sep 16 19:17:32.054530 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.054503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" (UID: "2376ac4a-f5bc-4068-a921-44c8fa3f9bc5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:32.054581 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.054530    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" (UID: "2376ac4a-f5bc-4068-a921-44c8fa3f9bc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:32.056315 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.056297    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" (UID: "2376ac4a-f5bc-4068-a921-44c8fa3f9bc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:32.076793 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.076757    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" (UID: "2376ac4a-f5bc-4068-a921-44c8fa3f9bc5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:32.099690 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.099658    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"]
Sep 16 19:17:32.104880 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.104853    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6cdb585d6-rh99x"]
Sep 16 19:17:32.155119 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.155080    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:32.155119 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.155109    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:32.155119 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.155123    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:32.155338 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.155137    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:17:32.780865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.780830    2578 generic.go:356] "Generic (PLEG): container finished" podID="2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" containerID="7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61" exitCode=0
Sep 16 19:17:32.781320 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.780877    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"
Sep 16 19:17:32.781320 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.780915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh" event={"ID":"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5","Type":"ContainerDied","Data":"7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61"}
Sep 16 19:17:32.781320 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.780965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh" event={"ID":"2376ac4a-f5bc-4068-a921-44c8fa3f9bc5","Type":"ContainerDied","Data":"08cf4f6c368cf38b1694b97a00fab85c315b6351cfad1ab7aa7cd4ad54f25602"}
Sep 16 19:17:32.781320 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.780992    2578 scope.go:117] "RemoveContainer" containerID="7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61"
Sep 16 19:17:32.789709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.789692    2578 scope.go:117] "RemoveContainer" containerID="acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a"
Sep 16 19:17:32.797717 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.797547    2578 scope.go:117] "RemoveContainer" containerID="7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61"
Sep 16 19:17:32.797836 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:17:32.797813    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61\": container with ID starting with 7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61 not found: ID does not exist" containerID="7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61"
Sep 16 19:17:32.797937 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.797838    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61"} err="failed to get container status \"7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61\": rpc error: code = NotFound desc = could not find container \"7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61\": container with ID starting with 7800bdb60a8867db9e1d950db023a8b0a38a6bf13686e4b0eb88aa37f99c1e61 not found: ID does not exist"
Sep 16 19:17:32.797937 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.797854    2578 scope.go:117] "RemoveContainer" containerID="acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a"
Sep 16 19:17:32.798113 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:17:32.798095    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a\": container with ID starting with acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a not found: ID does not exist" containerID="acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a"
Sep 16 19:17:32.798153 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.798119    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a"} err="failed to get container status \"acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a\": rpc error: code = NotFound desc = could not find container \"acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a\": container with ID starting with acecf1805dbc62e1cd966f4a9c5e3b5aa2a2d1b03d861c41f8d10450b61d066a not found: ID does not exist"
Sep 16 19:17:32.803948 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.803928    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"]
Sep 16 19:17:32.814658 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:32.814637    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-8657b7c9c6-b7pfh"]
Sep 16 19:17:33.064014 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:33.063935    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08c64343-7937-4070-bdff-24b222026360" path="/var/lib/kubelet/pods/08c64343-7937-4070-bdff-24b222026360/volumes"
Sep 16 19:17:33.064351 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:17:33.064338    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" path="/var/lib/kubelet/pods/2376ac4a-f5bc-4068-a921-44c8fa3f9bc5/volumes"
Sep 16 19:18:27.352668 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.352630    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"]
Sep 16 19:18:27.353267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353165    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c64343-7937-4070-bdff-24b222026360" containerName="storage-initializer"
Sep 16 19:18:27.353267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353186    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c64343-7937-4070-bdff-24b222026360" containerName="storage-initializer"
Sep 16 19:18:27.353267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353198    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" containerName="storage-initializer"
Sep 16 19:18:27.353267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353207    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" containerName="storage-initializer"
Sep 16 19:18:27.353267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353223    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" containerName="main"
Sep 16 19:18:27.353267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353233    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" containerName="main"
Sep 16 19:18:27.353267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353244    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" containerName="main"
Sep 16 19:18:27.353267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353251    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" containerName="main"
Sep 16 19:18:27.353726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353276    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c64343-7937-4070-bdff-24b222026360" containerName="main"
Sep 16 19:18:27.353726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353285    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c64343-7937-4070-bdff-24b222026360" containerName="main"
Sep 16 19:18:27.353726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353361    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c64343-7937-4070-bdff-24b222026360" containerName="main"
Sep 16 19:18:27.353726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353368    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c0d8ce-8e8a-40b1-9a37-5dc94d402bc2" containerName="main"
Sep 16 19:18:27.353726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.353376    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2376ac4a-f5bc-4068-a921-44c8fa3f9bc5" containerName="main"
Sep 16 19:18:27.356712 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.356689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.359554 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.359530    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 19:18:27.360671 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.360647    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-d2hjc\""
Sep 16 19:18:27.360815 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.360680    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 19:18:27.360815 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.360690    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 19:18:27.365945 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.365921    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"]
Sep 16 19:18:27.519099 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.519062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-home\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.519274 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.519109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.519274 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.519128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpchm\" (UniqueName: \"kubernetes.io/projected/e2620811-e255-42d0-a419-c8444bc5670d-kube-api-access-xpchm\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.519274 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.519240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2620811-e255-42d0-a419-c8444bc5670d-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.519381 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.519274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-model-cache\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.519381 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.519301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-dshm\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.619861 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.619779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-home\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.619861 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.619819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.619861 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.619838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpchm\" (UniqueName: \"kubernetes.io/projected/e2620811-e255-42d0-a419-c8444bc5670d-kube-api-access-xpchm\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.620121 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.619891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2620811-e255-42d0-a419-c8444bc5670d-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.620121 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.619917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-model-cache\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.620121 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.619949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-dshm\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.620363 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.620231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-home\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.620363 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.620255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.620363 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.620331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-model-cache\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.622183 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.622163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-dshm\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.622430 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.622413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2620811-e255-42d0-a419-c8444bc5670d-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.631847 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.631825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpchm\" (UniqueName: \"kubernetes.io/projected/e2620811-e255-42d0-a419-c8444bc5670d-kube-api-access-xpchm\") pod \"decom-v1-kserve-6cb797fb75-6jvf4\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.668344 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.668312    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:27.945348 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:27.945321    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"]
Sep 16 19:18:27.947361 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:18:27.947334    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2620811_e255_42d0_a419_c8444bc5670d.slice/crio-f9ae1f9923bdfc5e06d151be193d4d9a7740666e4c6787eb423b823dcf8a6cc0 WatchSource:0}: Error finding container f9ae1f9923bdfc5e06d151be193d4d9a7740666e4c6787eb423b823dcf8a6cc0: Status 404 returned error can't find the container with id f9ae1f9923bdfc5e06d151be193d4d9a7740666e4c6787eb423b823dcf8a6cc0
Sep 16 19:18:28.044338 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:28.044301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4" event={"ID":"e2620811-e255-42d0-a419-c8444bc5670d","Type":"ContainerStarted","Data":"68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b"}
Sep 16 19:18:28.044552 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:28.044375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4" event={"ID":"e2620811-e255-42d0-a419-c8444bc5670d","Type":"ContainerStarted","Data":"f9ae1f9923bdfc5e06d151be193d4d9a7740666e4c6787eb423b823dcf8a6cc0"}
Sep 16 19:18:28.066072 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:28.066016    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4" podStartSLOduration=1.066002115 podStartE2EDuration="1.066002115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:28.064642309 +0000 UTC m=+2015.511736855" watchObservedRunningTime="2026-09-16 19:18:28.066002115 +0000 UTC m=+2015.513096644"
Sep 16 19:18:28.668664 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:28.668631    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:37.669180 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:37.669124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:37.681546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:37.681522    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:37.966981 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:37.966896    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"]
Sep 16 19:18:37.970367 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:37.970351    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:37.972974 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:37.972946    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 16 19:18:37.984355 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:37.984333    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"]
Sep 16 19:18:38.100453 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.100420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:18:38.112146 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.112123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-dshm\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.112259 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.112163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddg6\" (UniqueName: \"kubernetes.io/projected/0d7a9ff8-151a-4c53-bc25-aafaff36484e-kube-api-access-jddg6\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.112259 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.112234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tls-certs\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.112337 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.112286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-home\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.112337 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.112304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-model-cache\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.112337 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.112323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tmp-dir\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.213804 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.213771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-dshm\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.213989 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.213824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddg6\" (UniqueName: \"kubernetes.io/projected/0d7a9ff8-151a-4c53-bc25-aafaff36484e-kube-api-access-jddg6\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.213989 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.213903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tls-certs\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.213989 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.213954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-home\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.213989 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.213976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-model-cache\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.214229 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.214130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tmp-dir\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.214326 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.214303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-home\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.214517 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.214460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tmp-dir\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.214604 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.214472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-model-cache\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.216021 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.216000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-dshm\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.216301 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.216281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tls-certs\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.222758 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.222706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddg6\" (UniqueName: \"kubernetes.io/projected/0d7a9ff8-151a-4c53-bc25-aafaff36484e-kube-api-access-jddg6\") pod \"decom-v2-kserve-77f867d787-g6m46\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.280578 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.280537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:38.553914 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:38.553889    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"]
Sep 16 19:18:38.556081 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:18:38.556054    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7a9ff8_151a_4c53_bc25_aafaff36484e.slice/crio-6975ee8e4980b6d12dcfee528017f809f9be6e6888aa73adb7e6d5abe6e78bc9 WatchSource:0}: Error finding container 6975ee8e4980b6d12dcfee528017f809f9be6e6888aa73adb7e6d5abe6e78bc9: Status 404 returned error can't find the container with id 6975ee8e4980b6d12dcfee528017f809f9be6e6888aa73adb7e6d5abe6e78bc9
Sep 16 19:18:39.102798 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:39.102756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46" event={"ID":"0d7a9ff8-151a-4c53-bc25-aafaff36484e","Type":"ContainerStarted","Data":"a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53"}
Sep 16 19:18:39.103294 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:39.102888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46" event={"ID":"0d7a9ff8-151a-4c53-bc25-aafaff36484e","Type":"ContainerStarted","Data":"6975ee8e4980b6d12dcfee528017f809f9be6e6888aa73adb7e6d5abe6e78bc9"}
Sep 16 19:18:39.122122 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:39.122065    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46" podStartSLOduration=2.122050762 podStartE2EDuration="2.122050762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:39.119958847 +0000 UTC m=+2026.567053396" watchObservedRunningTime="2026-09-16 19:18:39.122050762 +0000 UTC m=+2026.569145292"
Sep 16 19:18:39.280678 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:39.280645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:48.280991 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:48.280943    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:48.293812 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:48.293783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:49.158538 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.158510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:18:49.621121 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.621074    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"]
Sep 16 19:18:49.624825 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.624801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.627410 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.627385    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:18:49.627687 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.627670    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-qkkc7\""
Sep 16 19:18:49.628541 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.628521    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:18:49.628773 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.628756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:18:49.644139 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.644111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"]
Sep 16 19:18:49.704863 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.704827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdf026-850b-4b52-bbfc-06ab59b28735-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.705048 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.704892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.705048 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.704913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.705048 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.704943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1fdf026-850b-4b52-bbfc-06ab59b28735-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.806243 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.806197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1fdf026-850b-4b52-bbfc-06ab59b28735-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.806457 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.806263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdf026-850b-4b52-bbfc-06ab59b28735-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.806457 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.806343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.806623 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.806443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.806743 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.806723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.806829 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.806806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.806915 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.806898    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1fdf026-850b-4b52-bbfc-06ab59b28735-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.808824 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.808806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdf026-850b-4b52-bbfc-06ab59b28735-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:49.935858 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:49.935768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:50.217254 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:50.217163    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"]
Sep 16 19:18:50.220556 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:18:50.220521    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fdf026_850b_4b52_bbfc_06ab59b28735.slice/crio-95d4a18e5683e1d61df781e37e7110fb9fe89b32861ae038f6a44a238f17d4e3 WatchSource:0}: Error finding container 95d4a18e5683e1d61df781e37e7110fb9fe89b32861ae038f6a44a238f17d4e3: Status 404 returned error can't find the container with id 95d4a18e5683e1d61df781e37e7110fb9fe89b32861ae038f6a44a238f17d4e3
Sep 16 19:18:51.162431 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:51.162390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c" event={"ID":"f1fdf026-850b-4b52-bbfc-06ab59b28735","Type":"ContainerStarted","Data":"e92a6b8ab86cf3cfd48d0747dfece902817482b4a3eed7e20a4f4a71a32df2fd"}
Sep 16 19:18:51.162949 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:51.162460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c" event={"ID":"f1fdf026-850b-4b52-bbfc-06ab59b28735","Type":"ContainerStarted","Data":"95d4a18e5683e1d61df781e37e7110fb9fe89b32861ae038f6a44a238f17d4e3"}
Sep 16 19:18:53.173373 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:53.173340    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1fdf026-850b-4b52-bbfc-06ab59b28735" containerID="e92a6b8ab86cf3cfd48d0747dfece902817482b4a3eed7e20a4f4a71a32df2fd" exitCode=0
Sep 16 19:18:53.173809 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:53.173391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c" event={"ID":"f1fdf026-850b-4b52-bbfc-06ab59b28735","Type":"ContainerDied","Data":"e92a6b8ab86cf3cfd48d0747dfece902817482b4a3eed7e20a4f4a71a32df2fd"}
Sep 16 19:18:54.180396 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:54.180356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c" event={"ID":"f1fdf026-850b-4b52-bbfc-06ab59b28735","Type":"ContainerStarted","Data":"05e1da1da441fcf96668790a7087353d115db9fad0811ead5dd3d337385a86cf"}
Sep 16 19:18:54.180826 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:54.180673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:18:54.200166 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:18:54.200116    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c" podStartSLOduration=2.94269307 podStartE2EDuration="5.200100558s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.257407488s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:54.199443628 +0000 UTC m=+2041.646538185" watchObservedRunningTime="2026-09-16 19:18:54.200100558 +0000 UTC m=+2041.647195088"
Sep 16 19:19:25.196335 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:25.196298    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:19:27.482150 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:27.482118    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c_f1fdf026-850b-4b52-bbfc-06ab59b28735/main/0.log"
Sep 16 19:19:28.795409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:28.795368    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"]
Sep 16 19:19:28.795913 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:28.795722    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c" podUID="f1fdf026-850b-4b52-bbfc-06ab59b28735" containerName="main" containerID="cri-o://05e1da1da441fcf96668790a7087353d115db9fad0811ead5dd3d337385a86cf" gracePeriod=30
Sep 16 19:19:29.309272 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:19:29.309242    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fdf026_850b_4b52_bbfc_06ab59b28735.slice/crio-conmon-05e1da1da441fcf96668790a7087353d115db9fad0811ead5dd3d337385a86cf.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:19:29.369457 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.369425    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1fdf026-850b-4b52-bbfc-06ab59b28735" containerID="05e1da1da441fcf96668790a7087353d115db9fad0811ead5dd3d337385a86cf" exitCode=0
Sep 16 19:19:29.369618 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.369512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c" event={"ID":"f1fdf026-850b-4b52-bbfc-06ab59b28735","Type":"ContainerDied","Data":"05e1da1da441fcf96668790a7087353d115db9fad0811ead5dd3d337385a86cf"}
Sep 16 19:19:29.448883 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.448856    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:19:29.565091 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.565062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-kserve-provision-location\") pod \"f1fdf026-850b-4b52-bbfc-06ab59b28735\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") "
Sep 16 19:19:29.565290 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.565107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-tmp-dir\") pod \"f1fdf026-850b-4b52-bbfc-06ab59b28735\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") "
Sep 16 19:19:29.565290 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.565166    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdf026-850b-4b52-bbfc-06ab59b28735-tls-certs\") pod \"f1fdf026-850b-4b52-bbfc-06ab59b28735\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") "
Sep 16 19:19:29.565290 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.565214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1fdf026-850b-4b52-bbfc-06ab59b28735-cabundle-cert\") pod \"f1fdf026-850b-4b52-bbfc-06ab59b28735\" (UID: \"f1fdf026-850b-4b52-bbfc-06ab59b28735\") "
Sep 16 19:19:29.565605 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.565576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1fdf026-850b-4b52-bbfc-06ab59b28735" (UID: "f1fdf026-850b-4b52-bbfc-06ab59b28735"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.565675 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.565599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fdf026-850b-4b52-bbfc-06ab59b28735-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1fdf026-850b-4b52-bbfc-06ab59b28735" (UID: "f1fdf026-850b-4b52-bbfc-06ab59b28735"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:29.567438 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.567416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fdf026-850b-4b52-bbfc-06ab59b28735-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1fdf026-850b-4b52-bbfc-06ab59b28735" (UID: "f1fdf026-850b-4b52-bbfc-06ab59b28735"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:29.587921 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.587892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1fdf026-850b-4b52-bbfc-06ab59b28735" (UID: "f1fdf026-850b-4b52-bbfc-06ab59b28735"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.666447 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.666414    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdf026-850b-4b52-bbfc-06ab59b28735-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.666622 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.666466    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1fdf026-850b-4b52-bbfc-06ab59b28735-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.666622 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.666495    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.666622 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:29.666509    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdf026-850b-4b52-bbfc-06ab59b28735-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:30.376818 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:30.376787    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"
Sep 16 19:19:30.377314 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:30.376792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c" event={"ID":"f1fdf026-850b-4b52-bbfc-06ab59b28735","Type":"ContainerDied","Data":"95d4a18e5683e1d61df781e37e7110fb9fe89b32861ae038f6a44a238f17d4e3"}
Sep 16 19:19:30.377314 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:30.376919    2578 scope.go:117] "RemoveContainer" containerID="05e1da1da441fcf96668790a7087353d115db9fad0811ead5dd3d337385a86cf"
Sep 16 19:19:30.385538 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:30.385515    2578 scope.go:117] "RemoveContainer" containerID="e92a6b8ab86cf3cfd48d0747dfece902817482b4a3eed7e20a4f4a71a32df2fd"
Sep 16 19:19:30.401022 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:30.400994    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"]
Sep 16 19:19:30.408206 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:30.408183    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-58d9d88c84-dhq5c"]
Sep 16 19:19:31.064311 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:31.064276    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1fdf026-850b-4b52-bbfc-06ab59b28735" path="/var/lib/kubelet/pods/f1fdf026-850b-4b52-bbfc-06ab59b28735/volumes"
Sep 16 19:19:39.942195 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:39.942157    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"]
Sep 16 19:19:53.137528 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:53.137499    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:19:53.141627 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:53.141604    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:19:53.145569 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:53.145538    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:19:53.149598 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:53.149579    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:19:54.961439 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:54.961395    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46" podUID="0d7a9ff8-151a-4c53-bc25-aafaff36484e" containerName="main" containerID="cri-o://a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53" gracePeriod=45
Sep 16 19:19:55.210623 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.210597    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:19:55.280341 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jddg6\" (UniqueName: \"kubernetes.io/projected/0d7a9ff8-151a-4c53-bc25-aafaff36484e-kube-api-access-jddg6\") pod \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") "
Sep 16 19:19:55.280502 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280385    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tmp-dir\") pod \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") "
Sep 16 19:19:55.280502 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280410    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-home\") pod \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") "
Sep 16 19:19:55.280502 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tls-certs\") pod \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") "
Sep 16 19:19:55.280643 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280524    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-dshm\") pod \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") "
Sep 16 19:19:55.280643 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-model-cache\") pod \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\" (UID: \"0d7a9ff8-151a-4c53-bc25-aafaff36484e\") "
Sep 16 19:19:55.280743 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280686    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-home" (OuterVolumeSpecName: "home") pod "0d7a9ff8-151a-4c53-bc25-aafaff36484e" (UID: "0d7a9ff8-151a-4c53-bc25-aafaff36484e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:55.280743 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280699    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d7a9ff8-151a-4c53-bc25-aafaff36484e" (UID: "0d7a9ff8-151a-4c53-bc25-aafaff36484e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:55.280911 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280888    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d7a9ff8-151a-4c53-bc25-aafaff36484e" (UID: "0d7a9ff8-151a-4c53-bc25-aafaff36484e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:55.280911 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280892    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:55.281071 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.280936    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:55.282687 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.282662    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d7a9ff8-151a-4c53-bc25-aafaff36484e" (UID: "0d7a9ff8-151a-4c53-bc25-aafaff36484e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:55.282783 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.282764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-dshm" (OuterVolumeSpecName: "dshm") pod "0d7a9ff8-151a-4c53-bc25-aafaff36484e" (UID: "0d7a9ff8-151a-4c53-bc25-aafaff36484e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:55.282827 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.282788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7a9ff8-151a-4c53-bc25-aafaff36484e-kube-api-access-jddg6" (OuterVolumeSpecName: "kube-api-access-jddg6") pod "0d7a9ff8-151a-4c53-bc25-aafaff36484e" (UID: "0d7a9ff8-151a-4c53-bc25-aafaff36484e"). InnerVolumeSpecName "kube-api-access-jddg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:55.381639 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.381600    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jddg6\" (UniqueName: \"kubernetes.io/projected/0d7a9ff8-151a-4c53-bc25-aafaff36484e-kube-api-access-jddg6\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:55.381639 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.381633    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7a9ff8-151a-4c53-bc25-aafaff36484e-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:55.381639 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.381644    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:55.381875 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.381652    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7a9ff8-151a-4c53-bc25-aafaff36484e-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:19:55.507104 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.507059    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d7a9ff8-151a-4c53-bc25-aafaff36484e" containerID="a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53" exitCode=0
Sep 16 19:19:55.507280 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.507111    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"
Sep 16 19:19:55.507280 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.507125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46" event={"ID":"0d7a9ff8-151a-4c53-bc25-aafaff36484e","Type":"ContainerDied","Data":"a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53"}
Sep 16 19:19:55.507280 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.507178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46" event={"ID":"0d7a9ff8-151a-4c53-bc25-aafaff36484e","Type":"ContainerDied","Data":"6975ee8e4980b6d12dcfee528017f809f9be6e6888aa73adb7e6d5abe6e78bc9"}
Sep 16 19:19:55.507280 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.507209    2578 scope.go:117] "RemoveContainer" containerID="a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53"
Sep 16 19:19:55.516149 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.516129    2578 scope.go:117] "RemoveContainer" containerID="a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53"
Sep 16 19:19:55.516423 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:19:55.516396    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53\": container with ID starting with a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53 not found: ID does not exist" containerID="a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53"
Sep 16 19:19:55.516542 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.516429    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53"} err="failed to get container status \"a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53\": rpc error: code = NotFound desc = could not find container \"a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53\": container with ID starting with a766438662ebf7a0076603b559dc989393ba0a82534d4507531cf3b9c737da53 not found: ID does not exist"
Sep 16 19:19:55.534781 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.534714    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"]
Sep 16 19:19:55.542795 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:55.542768    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46"]
Sep 16 19:19:56.148152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:56.148106    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-g6m46" podUID="0d7a9ff8-151a-4c53-bc25-aafaff36484e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:19:56.276230 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:56.276195    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"]
Sep 16 19:19:57.063796 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:19:57.063763    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d7a9ff8-151a-4c53-bc25-aafaff36484e" path="/var/lib/kubelet/pods/0d7a9ff8-151a-4c53-bc25-aafaff36484e/volumes"
Sep 16 19:20:02.367951 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.367914    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"]
Sep 16 19:20:02.368416 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.368315    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1fdf026-850b-4b52-bbfc-06ab59b28735" containerName="storage-initializer"
Sep 16 19:20:02.368416 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.368331    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1fdf026-850b-4b52-bbfc-06ab59b28735" containerName="storage-initializer"
Sep 16 19:20:02.368416 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.368359    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7a9ff8-151a-4c53-bc25-aafaff36484e" containerName="main"
Sep 16 19:20:02.368416 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.368368    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7a9ff8-151a-4c53-bc25-aafaff36484e" containerName="main"
Sep 16 19:20:02.368416 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.368381    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1fdf026-850b-4b52-bbfc-06ab59b28735" containerName="main"
Sep 16 19:20:02.368416 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.368387    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1fdf026-850b-4b52-bbfc-06ab59b28735" containerName="main"
Sep 16 19:20:02.368693 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.368467    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1fdf026-850b-4b52-bbfc-06ab59b28735" containerName="main"
Sep 16 19:20:02.368693 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.368506    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7a9ff8-151a-4c53-bc25-aafaff36484e" containerName="main"
Sep 16 19:20:02.371551 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.371528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.375760 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.375739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-prkt7\""
Sep 16 19:20:02.375907 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.375754    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 16 19:20:02.375907 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.375765    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 19:20:02.375907 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.375789    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 19:20:02.388945 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.388905    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"]
Sep 16 19:20:02.443261 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.443216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-dshm\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.443436 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.443263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-model-cache\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.443436 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.443313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tls-certs\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.443436 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.443380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.443436 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.443404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-home\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.443630 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.443445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxxdq\" (UniqueName: \"kubernetes.io/projected/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-kube-api-access-jxxdq\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.544183 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.544145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-dshm\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.544183 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.544183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-model-cache\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.544456 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.544208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tls-certs\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.544456 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.544226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.544456 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.544243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-home\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.544456 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.544277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxxdq\" (UniqueName: \"kubernetes.io/projected/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-kube-api-access-jxxdq\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.544779 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.544610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-model-cache\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.544779 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.544711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.544779 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.544762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-home\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.546553 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.546534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-dshm\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.546831 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.546816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tls-certs\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.554144 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.554122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxxdq\" (UniqueName: \"kubernetes.io/projected/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-kube-api-access-jxxdq\") pod \"leave-v1-kserve-65684d4f5-gvsgn\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.681926 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.681828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:02.963267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:02.963187    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"]
Sep 16 19:20:02.966635 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:20:02.966608    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a963fd_eb7c_4f9d_8bb2_6bbb8a02e504.slice/crio-d6a4b984e5c269b3c23893ac32dc519657709752260befca82b7494755c1bea2 WatchSource:0}: Error finding container d6a4b984e5c269b3c23893ac32dc519657709752260befca82b7494755c1bea2: Status 404 returned error can't find the container with id d6a4b984e5c269b3c23893ac32dc519657709752260befca82b7494755c1bea2
Sep 16 19:20:03.545756 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:03.545718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn" event={"ID":"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504","Type":"ContainerStarted","Data":"809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee"}
Sep 16 19:20:03.546223 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:03.545778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn" event={"ID":"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504","Type":"ContainerStarted","Data":"d6a4b984e5c269b3c23893ac32dc519657709752260befca82b7494755c1bea2"}
Sep 16 19:20:03.574580 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:03.574524    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn" podStartSLOduration=1.5745056480000001 podStartE2EDuration="1.574505648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:03.571349889 +0000 UTC m=+2111.018444446" watchObservedRunningTime="2026-09-16 19:20:03.574505648 +0000 UTC m=+2111.021600175"
Sep 16 19:20:03.682508 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:03.682459    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:11.295439 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.295371    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4" podUID="e2620811-e255-42d0-a419-c8444bc5670d" containerName="main" containerID="cri-o://68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b" gracePeriod=45
Sep 16 19:20:11.541709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.541684    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:20:11.589865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.589767    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2620811-e255-42d0-a419-c8444bc5670d" containerID="68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b" exitCode=0
Sep 16 19:20:11.589865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.589812    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"
Sep 16 19:20:11.590093 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.589848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4" event={"ID":"e2620811-e255-42d0-a419-c8444bc5670d","Type":"ContainerDied","Data":"68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b"}
Sep 16 19:20:11.590093 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.589920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4" event={"ID":"e2620811-e255-42d0-a419-c8444bc5670d","Type":"ContainerDied","Data":"f9ae1f9923bdfc5e06d151be193d4d9a7740666e4c6787eb423b823dcf8a6cc0"}
Sep 16 19:20:11.590093 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.589950    2578 scope.go:117] "RemoveContainer" containerID="68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b"
Sep 16 19:20:11.598241 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.598221    2578 scope.go:117] "RemoveContainer" containerID="68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b"
Sep 16 19:20:11.598555 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:20:11.598525    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b\": container with ID starting with 68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b not found: ID does not exist" containerID="68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b"
Sep 16 19:20:11.598662 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.598556    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b"} err="failed to get container status \"68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b\": rpc error: code = NotFound desc = could not find container \"68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b\": container with ID starting with 68b46c6504277d8d66c46747303bfa1c1e27edaf0e1fb088adb702551de24a3b not found: ID does not exist"
Sep 16 19:20:11.621945 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.621912    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-home\") pod \"e2620811-e255-42d0-a419-c8444bc5670d\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") "
Sep 16 19:20:11.621945 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.621948    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-model-cache\") pod \"e2620811-e255-42d0-a419-c8444bc5670d\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") "
Sep 16 19:20:11.622175 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.621990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-dshm\") pod \"e2620811-e255-42d0-a419-c8444bc5670d\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") "
Sep 16 19:20:11.622175 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.622049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpchm\" (UniqueName: \"kubernetes.io/projected/e2620811-e255-42d0-a419-c8444bc5670d-kube-api-access-xpchm\") pod \"e2620811-e255-42d0-a419-c8444bc5670d\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") "
Sep 16 19:20:11.622175 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.622086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-tmp-dir\") pod \"e2620811-e255-42d0-a419-c8444bc5670d\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") "
Sep 16 19:20:11.622175 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.622118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2620811-e255-42d0-a419-c8444bc5670d-tls-certs\") pod \"e2620811-e255-42d0-a419-c8444bc5670d\" (UID: \"e2620811-e255-42d0-a419-c8444bc5670d\") "
Sep 16 19:20:11.622359 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.622220    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2620811-e255-42d0-a419-c8444bc5670d" (UID: "e2620811-e255-42d0-a419-c8444bc5670d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:11.622359 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.622241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-home" (OuterVolumeSpecName: "home") pod "e2620811-e255-42d0-a419-c8444bc5670d" (UID: "e2620811-e255-42d0-a419-c8444bc5670d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:11.622443 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.622406    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.622443 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.622422    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.622443 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.622426    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2620811-e255-42d0-a419-c8444bc5670d" (UID: "e2620811-e255-42d0-a419-c8444bc5670d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:11.624198 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.624168    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-dshm" (OuterVolumeSpecName: "dshm") pod "e2620811-e255-42d0-a419-c8444bc5670d" (UID: "e2620811-e255-42d0-a419-c8444bc5670d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:11.624198 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.624170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2620811-e255-42d0-a419-c8444bc5670d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2620811-e255-42d0-a419-c8444bc5670d" (UID: "e2620811-e255-42d0-a419-c8444bc5670d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:11.624338 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.624246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2620811-e255-42d0-a419-c8444bc5670d-kube-api-access-xpchm" (OuterVolumeSpecName: "kube-api-access-xpchm") pod "e2620811-e255-42d0-a419-c8444bc5670d" (UID: "e2620811-e255-42d0-a419-c8444bc5670d"). InnerVolumeSpecName "kube-api-access-xpchm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:11.723470 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.723428    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.723470 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.723461    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2620811-e255-42d0-a419-c8444bc5670d-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.723470 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.723470    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2620811-e255-42d0-a419-c8444bc5670d-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.723745 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.723504    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpchm\" (UniqueName: \"kubernetes.io/projected/e2620811-e255-42d0-a419-c8444bc5670d-kube-api-access-xpchm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:20:11.912622 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.912588    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"]
Sep 16 19:20:11.916823 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:11.916799    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-6jvf4"]
Sep 16 19:20:12.682011 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:12.681979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:12.694562 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:12.694531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:13.065643 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:13.065608    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2620811-e255-42d0-a419-c8444bc5670d" path="/var/lib/kubelet/pods/e2620811-e255-42d0-a419-c8444bc5670d/volumes"
Sep 16 19:20:13.613030 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:13.612999    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:20:41.096786 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.096748    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"]
Sep 16 19:20:41.097258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.097091    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2620811-e255-42d0-a419-c8444bc5670d" containerName="main"
Sep 16 19:20:41.097258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.097102    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2620811-e255-42d0-a419-c8444bc5670d" containerName="main"
Sep 16 19:20:41.097258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.097174    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2620811-e255-42d0-a419-c8444bc5670d" containerName="main"
Sep 16 19:20:41.100106 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.100088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.109308 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.109281    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:20:41.109435 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.109309    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-xwg66\""
Sep 16 19:20:41.109435 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.109309    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:20:41.110178 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.110164    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:20:41.127217 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.127188    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"]
Sep 16 19:20:41.179674 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.179639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.179839 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.179688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.179839 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.179729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tgdq\" (UniqueName: \"kubernetes.io/projected/961385fc-3cb1-490d-adca-b12b17f159cb-kube-api-access-8tgdq\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.179839 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.179806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/961385fc-3cb1-490d-adca-b12b17f159cb-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.179839 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.179826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.179985 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.179856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.280435 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.280395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.280435 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.280437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tgdq\" (UniqueName: \"kubernetes.io/projected/961385fc-3cb1-490d-adca-b12b17f159cb-kube-api-access-8tgdq\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.280709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.280471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/961385fc-3cb1-490d-adca-b12b17f159cb-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.280709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.280513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.280709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.280551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.280709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.280619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.280929 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.280879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.280929 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.280915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.281016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.280941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.282846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.282828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.283032 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.283013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/961385fc-3cb1-490d-adca-b12b17f159cb-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.290926 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.290899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tgdq\" (UniqueName: \"kubernetes.io/projected/961385fc-3cb1-490d-adca-b12b17f159cb-kube-api-access-8tgdq\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.410211 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.410115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:41.691036 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.688532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"]
Sep 16 19:20:41.741112 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:41.741076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z" event={"ID":"961385fc-3cb1-490d-adca-b12b17f159cb","Type":"ContainerStarted","Data":"64b8a23956b778fe6235d873dbcb703e5f7c2d30cbcb260d4d6efbff24af109e"}
Sep 16 19:20:42.747641 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:42.747604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z" event={"ID":"961385fc-3cb1-490d-adca-b12b17f159cb","Type":"ContainerStarted","Data":"360364ef84cb3e8b061095b954e2585b48fc66821268a8256a67af5a97aa49a3"}
Sep 16 19:20:42.768606 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:42.768547    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z" podStartSLOduration=1.768532352 podStartE2EDuration="1.768532352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:42.767107112 +0000 UTC m=+2150.214201669" watchObservedRunningTime="2026-09-16 19:20:42.768532352 +0000 UTC m=+2150.215626881"
Sep 16 19:20:43.410423 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:43.410380    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:51.411081 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:51.411044    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:51.423202 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:51.423173    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:20:51.804421 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:20:51.804385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:21:39.695355 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:39.695320    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"]
Sep 16 19:21:39.704807 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:21:39.704770    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:21:39.704807 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:39.704807 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:39.704807 ip-10-0-143-161 kubenswrapper[2578]:  > containerID="360364ef84cb3e8b061095b954e2585b48fc66821268a8256a67af5a97aa49a3" cmd=["/bin/sleep","15"]
Sep 16 19:21:39.705074 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:39.704828    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z" message="" err=<
Sep 16 19:21:39.705074 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:39.705074 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:39.705074 ip-10-0-143-161 kubenswrapper[2578]:  >
Sep 16 19:21:39.705074 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:21:39.704841    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:21:39.705074 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:39.705074 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:39.705074 ip-10-0-143-161 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z" podUID="961385fc-3cb1-490d-adca-b12b17f159cb" containerName="main" containerID="cri-o://360364ef84cb3e8b061095b954e2585b48fc66821268a8256a67af5a97aa49a3"
Sep 16 19:21:39.705074 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:39.704890    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z" podUID="961385fc-3cb1-490d-adca-b12b17f159cb" containerName="main" containerID="cri-o://360364ef84cb3e8b061095b954e2585b48fc66821268a8256a67af5a97aa49a3" gracePeriod=60
Sep 16 19:21:39.793796 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:39.793753    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z" podUID="961385fc-3cb1-490d-adca-b12b17f159cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 16 19:21:40.034333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.034298    2578 generic.go:356] "Generic (PLEG): container finished" podID="961385fc-3cb1-490d-adca-b12b17f159cb" containerID="360364ef84cb3e8b061095b954e2585b48fc66821268a8256a67af5a97aa49a3" exitCode=0
Sep 16 19:21:40.034465 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.034366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z" event={"ID":"961385fc-3cb1-490d-adca-b12b17f159cb","Type":"ContainerDied","Data":"360364ef84cb3e8b061095b954e2585b48fc66821268a8256a67af5a97aa49a3"}
Sep 16 19:21:40.052681 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.052656    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:21:40.092618 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.092584    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/961385fc-3cb1-490d-adca-b12b17f159cb-tls-certs\") pod \"961385fc-3cb1-490d-adca-b12b17f159cb\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") "
Sep 16 19:21:40.092784 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.092667    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-dshm\") pod \"961385fc-3cb1-490d-adca-b12b17f159cb\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") "
Sep 16 19:21:40.092784 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.092728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-model-cache\") pod \"961385fc-3cb1-490d-adca-b12b17f159cb\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") "
Sep 16 19:21:40.092784 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.092765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tgdq\" (UniqueName: \"kubernetes.io/projected/961385fc-3cb1-490d-adca-b12b17f159cb-kube-api-access-8tgdq\") pod \"961385fc-3cb1-490d-adca-b12b17f159cb\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") "
Sep 16 19:21:40.092969 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.092809    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-tmp-dir\") pod \"961385fc-3cb1-490d-adca-b12b17f159cb\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") "
Sep 16 19:21:40.092969 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.092838    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-home\") pod \"961385fc-3cb1-490d-adca-b12b17f159cb\" (UID: \"961385fc-3cb1-490d-adca-b12b17f159cb\") "
Sep 16 19:21:40.093256 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.093207    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "961385fc-3cb1-490d-adca-b12b17f159cb" (UID: "961385fc-3cb1-490d-adca-b12b17f159cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:40.093256 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.093230    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-home" (OuterVolumeSpecName: "home") pod "961385fc-3cb1-490d-adca-b12b17f159cb" (UID: "961385fc-3cb1-490d-adca-b12b17f159cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:40.093256 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.093218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "961385fc-3cb1-490d-adca-b12b17f159cb" (UID: "961385fc-3cb1-490d-adca-b12b17f159cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:40.095178 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.095149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961385fc-3cb1-490d-adca-b12b17f159cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "961385fc-3cb1-490d-adca-b12b17f159cb" (UID: "961385fc-3cb1-490d-adca-b12b17f159cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:40.095178 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.095172    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-dshm" (OuterVolumeSpecName: "dshm") pod "961385fc-3cb1-490d-adca-b12b17f159cb" (UID: "961385fc-3cb1-490d-adca-b12b17f159cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:40.095346 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.095161    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961385fc-3cb1-490d-adca-b12b17f159cb-kube-api-access-8tgdq" (OuterVolumeSpecName: "kube-api-access-8tgdq") pod "961385fc-3cb1-490d-adca-b12b17f159cb" (UID: "961385fc-3cb1-490d-adca-b12b17f159cb"). InnerVolumeSpecName "kube-api-access-8tgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:40.193543 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.193502    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:21:40.193543 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.193536    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:21:40.193543 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.193548    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tgdq\" (UniqueName: \"kubernetes.io/projected/961385fc-3cb1-490d-adca-b12b17f159cb-kube-api-access-8tgdq\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:21:40.193792 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.193558    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:21:40.193792 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.193567    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/961385fc-3cb1-490d-adca-b12b17f159cb-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:21:40.193792 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:40.193575    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/961385fc-3cb1-490d-adca-b12b17f159cb-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:21:41.040576 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:41.040546    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"
Sep 16 19:21:41.041040 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:41.040580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z" event={"ID":"961385fc-3cb1-490d-adca-b12b17f159cb","Type":"ContainerDied","Data":"64b8a23956b778fe6235d873dbcb703e5f7c2d30cbcb260d4d6efbff24af109e"}
Sep 16 19:21:41.041040 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:41.040651    2578 scope.go:117] "RemoveContainer" containerID="360364ef84cb3e8b061095b954e2585b48fc66821268a8256a67af5a97aa49a3"
Sep 16 19:21:41.072384 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:41.072353    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"]
Sep 16 19:21:41.084537 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:41.084509    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-bk27z"]
Sep 16 19:21:43.063634 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:43.063602    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="961385fc-3cb1-490d-adca-b12b17f159cb" path="/var/lib/kubelet/pods/961385fc-3cb1-490d-adca-b12b17f159cb/volumes"
Sep 16 19:21:45.819051 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:21:45.819011    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"]
Sep 16 19:22:00.838522 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:00.838458    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn" podUID="99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" containerName="main" containerID="cri-o://809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee" gracePeriod=45
Sep 16 19:22:01.095701 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.095635    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:22:01.136424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.136385    2578 generic.go:356] "Generic (PLEG): container finished" podID="99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" containerID="809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee" exitCode=0
Sep 16 19:22:01.136751 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.136442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"
Sep 16 19:22:01.136751 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.136463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn" event={"ID":"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504","Type":"ContainerDied","Data":"809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee"}
Sep 16 19:22:01.136751 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.136579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn" event={"ID":"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504","Type":"ContainerDied","Data":"d6a4b984e5c269b3c23893ac32dc519657709752260befca82b7494755c1bea2"}
Sep 16 19:22:01.136751 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.136608    2578 scope.go:117] "RemoveContainer" containerID="809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee"
Sep 16 19:22:01.145405 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.145387    2578 scope.go:117] "RemoveContainer" containerID="809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee"
Sep 16 19:22:01.145722 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:22:01.145694    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee\": container with ID starting with 809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee not found: ID does not exist" containerID="809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee"
Sep 16 19:22:01.145787 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.145733    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee"} err="failed to get container status \"809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee\": rpc error: code = NotFound desc = could not find container \"809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee\": container with ID starting with 809c3ccb4d35c1541c5c279278e859576a9b6c3bcf5c9ecdb9847114297267ee not found: ID does not exist"
Sep 16 19:22:01.178174 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178133    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tmp-dir\") pod \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") "
Sep 16 19:22:01.178352 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-home\") pod \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") "
Sep 16 19:22:01.178352 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tls-certs\") pod \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") "
Sep 16 19:22:01.178352 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178275    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-dshm\") pod \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") "
Sep 16 19:22:01.178352 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxxdq\" (UniqueName: \"kubernetes.io/projected/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-kube-api-access-jxxdq\") pod \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") "
Sep 16 19:22:01.178601 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-model-cache\") pod \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\" (UID: \"99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504\") "
Sep 16 19:22:01.178601 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" (UID: "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:01.178601 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178545    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-home" (OuterVolumeSpecName: "home") pod "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" (UID: "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:01.178712 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178696    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:22:01.178755 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178714    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-model-cache" (OuterVolumeSpecName: "model-cache") pod "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" (UID: "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:01.178755 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.178736    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:22:01.180734 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.180710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" (UID: "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:01.180858 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.180835    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-dshm" (OuterVolumeSpecName: "dshm") pod "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" (UID: "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:01.180935 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.180840    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-kube-api-access-jxxdq" (OuterVolumeSpecName: "kube-api-access-jxxdq") pod "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" (UID: "99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504"). InnerVolumeSpecName "kube-api-access-jxxdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:01.279206 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.279167    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:22:01.279206 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.279198    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:22:01.279206 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.279211    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxxdq\" (UniqueName: \"kubernetes.io/projected/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-kube-api-access-jxxdq\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:22:01.279451 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.279224    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:22:01.460463 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.460427    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"]
Sep 16 19:22:01.465882 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.465857    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-gvsgn"]
Sep 16 19:22:01.880378 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.880338    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"]
Sep 16 19:22:01.880783 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.880768    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="961385fc-3cb1-490d-adca-b12b17f159cb" containerName="main"
Sep 16 19:22:01.880828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.880785    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="961385fc-3cb1-490d-adca-b12b17f159cb" containerName="main"
Sep 16 19:22:01.880828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.880794    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" containerName="main"
Sep 16 19:22:01.880828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.880799    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" containerName="main"
Sep 16 19:22:01.880966 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.880861    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" containerName="main"
Sep 16 19:22:01.880966 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.880871    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="961385fc-3cb1-490d-adca-b12b17f159cb" containerName="main"
Sep 16 19:22:01.885244 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.885219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:01.889734 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.889706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 16 19:22:01.889857 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.889706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-clqmt\""
Sep 16 19:22:01.889857 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.889787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 19:22:01.889857 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.889711    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 19:22:01.905611 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.905588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"]
Sep 16 19:22:01.985393 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.985347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-home\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:01.985393 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.985399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:01.985651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.985547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-dshm\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:01.985651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.985584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:01.985651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.985600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6frt\" (UniqueName: \"kubernetes.io/projected/857e7fa5-5966-438c-b980-e29b8f93cad8-kube-api-access-l6frt\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:01.985651 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:01.985621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857e7fa5-5966-438c-b980-e29b8f93cad8-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.086717 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.086678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-dshm\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.086717 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.086718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.086978 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.086738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6frt\" (UniqueName: \"kubernetes.io/projected/857e7fa5-5966-438c-b980-e29b8f93cad8-kube-api-access-l6frt\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.086978 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.086756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857e7fa5-5966-438c-b980-e29b8f93cad8-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.086978 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.086941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-home\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.087136 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.086989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.087175 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.087149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.087264 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.087248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-home\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.087320 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.087301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.089026 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.089003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-dshm\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.089221 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.089206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857e7fa5-5966-438c-b980-e29b8f93cad8-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.096757 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.096731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6frt\" (UniqueName: \"kubernetes.io/projected/857e7fa5-5966-438c-b980-e29b8f93cad8-kube-api-access-l6frt\") pod \"tri-v1-kserve-79f8f98bc5-dgmwr\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.195336 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.195241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:02.469340 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:02.469314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"]
Sep 16 19:22:02.471493 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:22:02.471451    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857e7fa5_5966_438c_b980_e29b8f93cad8.slice/crio-94902d155c5e86907c326b228a18e11545552a729aa8a7310f85d05d41328887 WatchSource:0}: Error finding container 94902d155c5e86907c326b228a18e11545552a729aa8a7310f85d05d41328887: Status 404 returned error can't find the container with id 94902d155c5e86907c326b228a18e11545552a729aa8a7310f85d05d41328887
Sep 16 19:22:03.064353 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:03.064314    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504" path="/var/lib/kubelet/pods/99a963fd-eb7c-4f9d-8bb2-6bbb8a02e504/volumes"
Sep 16 19:22:03.149530 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:03.149462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr" event={"ID":"857e7fa5-5966-438c-b980-e29b8f93cad8","Type":"ContainerStarted","Data":"3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418"}
Sep 16 19:22:03.149700 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:03.149535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr" event={"ID":"857e7fa5-5966-438c-b980-e29b8f93cad8","Type":"ContainerStarted","Data":"94902d155c5e86907c326b228a18e11545552a729aa8a7310f85d05d41328887"}
Sep 16 19:22:03.187721 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:03.187668    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr" podStartSLOduration=2.187653479 podStartE2EDuration="2.187653479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:03.187311192 +0000 UTC m=+2230.634405744" watchObservedRunningTime="2026-09-16 19:22:03.187653479 +0000 UTC m=+2230.634748009"
Sep 16 19:22:03.195327 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:03.195301    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:12.195565 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:12.195524    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:12.208076 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:12.208047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:13.206676 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:13.206645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:22:19.552024 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.551946    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"]
Sep 16 19:22:19.555760 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.555736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.559437 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.559418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 16 19:22:19.574443 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.574418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"]
Sep 16 19:22:19.739227 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.739192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-model-cache\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.739427 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.739251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.739427 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.739362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27lsn\" (UniqueName: \"kubernetes.io/projected/52327db1-afe1-4be9-bbf2-7dd91851308d-kube-api-access-27lsn\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.739427 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.739409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-dshm\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.739608 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.739429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-home\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.739608 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.739448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52327db1-afe1-4be9-bbf2-7dd91851308d-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.840248 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.840144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-dshm\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.840248 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.840196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-home\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.840248 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.840214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52327db1-afe1-4be9-bbf2-7dd91851308d-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.840248 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.840243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-model-cache\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.840572 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.840280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.840572 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.840340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27lsn\" (UniqueName: \"kubernetes.io/projected/52327db1-afe1-4be9-bbf2-7dd91851308d-kube-api-access-27lsn\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.840736 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.840709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-model-cache\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.840805 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.840723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-home\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.840805 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.840785    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.842689 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.842668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-dshm\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.842907 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.842886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52327db1-afe1-4be9-bbf2-7dd91851308d-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.851217 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.851190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27lsn\" (UniqueName: \"kubernetes.io/projected/52327db1-afe1-4be9-bbf2-7dd91851308d-kube-api-access-27lsn\") pod \"tri-v2-kserve-dcb554bf5-w2zmg\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:19.865905 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:19.865872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:20.160752 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:20.160725    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"]
Sep 16 19:22:20.163053 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:22:20.163022    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52327db1_afe1_4be9_bbf2_7dd91851308d.slice/crio-8de22d7606b3207e41b0458a41bdd5126a4637b6c52e4b18a5322fee293a0cac WatchSource:0}: Error finding container 8de22d7606b3207e41b0458a41bdd5126a4637b6c52e4b18a5322fee293a0cac: Status 404 returned error can't find the container with id 8de22d7606b3207e41b0458a41bdd5126a4637b6c52e4b18a5322fee293a0cac
Sep 16 19:22:20.230984 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:20.230960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg" event={"ID":"52327db1-afe1-4be9-bbf2-7dd91851308d","Type":"ContainerStarted","Data":"8de22d7606b3207e41b0458a41bdd5126a4637b6c52e4b18a5322fee293a0cac"}
Sep 16 19:22:21.236989 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:21.236949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg" event={"ID":"52327db1-afe1-4be9-bbf2-7dd91851308d","Type":"ContainerStarted","Data":"e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff"}
Sep 16 19:22:21.263428 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:21.263376    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg" podStartSLOduration=2.2633576140000002 podStartE2EDuration="2.263357614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:21.262674223 +0000 UTC m=+2248.709768788" watchObservedRunningTime="2026-09-16 19:22:21.263357614 +0000 UTC m=+2248.710452144"
Sep 16 19:22:21.866282 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:21.866243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:29.866146 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:29.866105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:29.878618 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:29.878590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:22:30.293740 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:22:30.293707    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:24:15.285677 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:15.285641    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"]
Sep 16 19:24:15.293984 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:15.293953    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"]
Sep 16 19:24:26.555082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.555037    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"]
Sep 16 19:24:26.559120 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.559093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.562077 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.562054    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:24:26.563090 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.563068    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:24:26.563224 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.563202    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:24:26.563296 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.563214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-j57wq\""
Sep 16 19:24:26.570629 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.570607    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"]
Sep 16 19:24:26.654332 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.654297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.654332 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.654333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.654580 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.654405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.654580 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.654461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.654580 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.654504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv2dp\" (UniqueName: \"kubernetes.io/projected/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-kube-api-access-sv2dp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.654580 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.654543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.755534 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.755497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.755736 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.755541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.755736 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.755588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.755736 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.755638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.755736 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.755664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv2dp\" (UniqueName: \"kubernetes.io/projected/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-kube-api-access-sv2dp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.755982 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.755863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.756047 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.756014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.756047 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.756019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.756150 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.756063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.758142 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.758121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.758254 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.758222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.765062 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.765041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv2dp\" (UniqueName: \"kubernetes.io/projected/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-kube-api-access-sv2dp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:26.871263 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:26.871174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:27.151588 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:27.151299    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"]
Sep 16 19:24:27.853268 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:27.853227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8" event={"ID":"17c267a1-b977-459f-b71f-ae7b8c4f3c3c","Type":"ContainerStarted","Data":"f8c4f75be6cdd056ec341dc59684aeabb05ae31fdeece00084fbe18b7bee7b5f"}
Sep 16 19:24:27.853762 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:27.853281    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8" event={"ID":"17c267a1-b977-459f-b71f-ae7b8c4f3c3c","Type":"ContainerStarted","Data":"9883bef5ae48f3a0b2e066be2fcceca911986808408de4b2edfed52e486b9a67"}
Sep 16 19:24:27.871863 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:27.871834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:27.874789 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:27.874751    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8" podStartSLOduration=1.874738062 podStartE2EDuration="1.874738062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:27.873125474 +0000 UTC m=+2375.320220021" watchObservedRunningTime="2026-09-16 19:24:27.874738062 +0000 UTC m=+2375.321832588"
Sep 16 19:24:30.306131 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.306091    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr" podUID="857e7fa5-5966-438c-b980-e29b8f93cad8" containerName="main" containerID="cri-o://3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418" gracePeriod=45
Sep 16 19:24:30.350192 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.350152    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg" podUID="52327db1-afe1-4be9-bbf2-7dd91851308d" containerName="main" containerID="cri-o://e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff" gracePeriod=45
Sep 16 19:24:30.352704 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:24:30.352674    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857e7fa5_5966_438c_b980_e29b8f93cad8.slice/crio-conmon-3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:24:30.617696 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.617351    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:24:30.621106 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.621078    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:24:30.695440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695406    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52327db1-afe1-4be9-bbf2-7dd91851308d-tls-certs\") pod \"52327db1-afe1-4be9-bbf2-7dd91851308d\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") "
Sep 16 19:24:30.695652 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857e7fa5-5966-438c-b980-e29b8f93cad8-tls-certs\") pod \"857e7fa5-5966-438c-b980-e29b8f93cad8\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") "
Sep 16 19:24:30.695652 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-dshm\") pod \"857e7fa5-5966-438c-b980-e29b8f93cad8\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") "
Sep 16 19:24:30.695781 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695678    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-home\") pod \"857e7fa5-5966-438c-b980-e29b8f93cad8\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") "
Sep 16 19:24:30.695781 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-tmp-dir\") pod \"857e7fa5-5966-438c-b980-e29b8f93cad8\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") "
Sep 16 19:24:30.695781 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27lsn\" (UniqueName: \"kubernetes.io/projected/52327db1-afe1-4be9-bbf2-7dd91851308d-kube-api-access-27lsn\") pod \"52327db1-afe1-4be9-bbf2-7dd91851308d\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") "
Sep 16 19:24:30.695949 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-home\") pod \"52327db1-afe1-4be9-bbf2-7dd91851308d\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") "
Sep 16 19:24:30.695949 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695881    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-model-cache\") pod \"52327db1-afe1-4be9-bbf2-7dd91851308d\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") "
Sep 16 19:24:30.695949 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-dshm\") pod \"52327db1-afe1-4be9-bbf2-7dd91851308d\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") "
Sep 16 19:24:30.695949 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695936    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-tmp-dir\") pod \"52327db1-afe1-4be9-bbf2-7dd91851308d\" (UID: \"52327db1-afe1-4be9-bbf2-7dd91851308d\") "
Sep 16 19:24:30.696165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.695973    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6frt\" (UniqueName: \"kubernetes.io/projected/857e7fa5-5966-438c-b980-e29b8f93cad8-kube-api-access-l6frt\") pod \"857e7fa5-5966-438c-b980-e29b8f93cad8\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") "
Sep 16 19:24:30.696165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.696018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-model-cache\") pod \"857e7fa5-5966-438c-b980-e29b8f93cad8\" (UID: \"857e7fa5-5966-438c-b980-e29b8f93cad8\") "
Sep 16 19:24:30.696165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.696101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "857e7fa5-5966-438c-b980-e29b8f93cad8" (UID: "857e7fa5-5966-438c-b980-e29b8f93cad8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.696322 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.696296    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-model-cache" (OuterVolumeSpecName: "model-cache") pod "857e7fa5-5966-438c-b980-e29b8f93cad8" (UID: "857e7fa5-5966-438c-b980-e29b8f93cad8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.696726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.696424    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.696726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.696469    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.696726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.696513    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-model-cache" (OuterVolumeSpecName: "model-cache") pod "52327db1-afe1-4be9-bbf2-7dd91851308d" (UID: "52327db1-afe1-4be9-bbf2-7dd91851308d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.696726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.696579    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-home" (OuterVolumeSpecName: "home") pod "857e7fa5-5966-438c-b980-e29b8f93cad8" (UID: "857e7fa5-5966-438c-b980-e29b8f93cad8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.696726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.696655    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-home" (OuterVolumeSpecName: "home") pod "52327db1-afe1-4be9-bbf2-7dd91851308d" (UID: "52327db1-afe1-4be9-bbf2-7dd91851308d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.696726 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.696674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52327db1-afe1-4be9-bbf2-7dd91851308d" (UID: "52327db1-afe1-4be9-bbf2-7dd91851308d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.698408 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.698386    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-dshm" (OuterVolumeSpecName: "dshm") pod "857e7fa5-5966-438c-b980-e29b8f93cad8" (UID: "857e7fa5-5966-438c-b980-e29b8f93cad8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.698694 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.698668    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857e7fa5-5966-438c-b980-e29b8f93cad8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "857e7fa5-5966-438c-b980-e29b8f93cad8" (UID: "857e7fa5-5966-438c-b980-e29b8f93cad8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:30.698805 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.698690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52327db1-afe1-4be9-bbf2-7dd91851308d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52327db1-afe1-4be9-bbf2-7dd91851308d" (UID: "52327db1-afe1-4be9-bbf2-7dd91851308d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:30.698960 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.698930    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-dshm" (OuterVolumeSpecName: "dshm") pod "52327db1-afe1-4be9-bbf2-7dd91851308d" (UID: "52327db1-afe1-4be9-bbf2-7dd91851308d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:30.699067 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.699026    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52327db1-afe1-4be9-bbf2-7dd91851308d-kube-api-access-27lsn" (OuterVolumeSpecName: "kube-api-access-27lsn") pod "52327db1-afe1-4be9-bbf2-7dd91851308d" (UID: "52327db1-afe1-4be9-bbf2-7dd91851308d"). InnerVolumeSpecName "kube-api-access-27lsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:30.699708 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.699688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857e7fa5-5966-438c-b980-e29b8f93cad8-kube-api-access-l6frt" (OuterVolumeSpecName: "kube-api-access-l6frt") pod "857e7fa5-5966-438c-b980-e29b8f93cad8" (UID: "857e7fa5-5966-438c-b980-e29b8f93cad8"). InnerVolumeSpecName "kube-api-access-l6frt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:30.797287 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797257    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.797287 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797284    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.797287 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797294    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.797546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797303    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6frt\" (UniqueName: \"kubernetes.io/projected/857e7fa5-5966-438c-b980-e29b8f93cad8-kube-api-access-l6frt\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.797546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797312    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52327db1-afe1-4be9-bbf2-7dd91851308d-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.797546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797323    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857e7fa5-5966-438c-b980-e29b8f93cad8-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.797546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797330    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.797546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797337    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857e7fa5-5966-438c-b980-e29b8f93cad8-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.797546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797344    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27lsn\" (UniqueName: \"kubernetes.io/projected/52327db1-afe1-4be9-bbf2-7dd91851308d-kube-api-access-27lsn\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.797546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.797353    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52327db1-afe1-4be9-bbf2-7dd91851308d-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:24:30.868708 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.868616    2578 generic.go:356] "Generic (PLEG): container finished" podID="52327db1-afe1-4be9-bbf2-7dd91851308d" containerID="e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff" exitCode=0
Sep 16 19:24:30.868708 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.868667    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"
Sep 16 19:24:30.868930 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.868713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg" event={"ID":"52327db1-afe1-4be9-bbf2-7dd91851308d","Type":"ContainerDied","Data":"e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff"}
Sep 16 19:24:30.868930 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.868783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg" event={"ID":"52327db1-afe1-4be9-bbf2-7dd91851308d","Type":"ContainerDied","Data":"8de22d7606b3207e41b0458a41bdd5126a4637b6c52e4b18a5322fee293a0cac"}
Sep 16 19:24:30.868930 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.868816    2578 scope.go:117] "RemoveContainer" containerID="e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff"
Sep 16 19:24:30.870082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.870042    2578 generic.go:356] "Generic (PLEG): container finished" podID="857e7fa5-5966-438c-b980-e29b8f93cad8" containerID="3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418" exitCode=0
Sep 16 19:24:30.870082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.870072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr" event={"ID":"857e7fa5-5966-438c-b980-e29b8f93cad8","Type":"ContainerDied","Data":"3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418"}
Sep 16 19:24:30.870218 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.870104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr" event={"ID":"857e7fa5-5966-438c-b980-e29b8f93cad8","Type":"ContainerDied","Data":"94902d155c5e86907c326b228a18e11545552a729aa8a7310f85d05d41328887"}
Sep 16 19:24:30.870218 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.870103    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"
Sep 16 19:24:30.878905 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.878888    2578 scope.go:117] "RemoveContainer" containerID="e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff"
Sep 16 19:24:30.879163 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:24:30.879140    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff\": container with ID starting with e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff not found: ID does not exist" containerID="e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff"
Sep 16 19:24:30.879214 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.879172    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff"} err="failed to get container status \"e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff\": rpc error: code = NotFound desc = could not find container \"e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff\": container with ID starting with e8e4a23267e892499c6d75034745b2b6f2ccd9682ae79ba7cd679d0d82387bff not found: ID does not exist"
Sep 16 19:24:30.879214 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.879190    2578 scope.go:117] "RemoveContainer" containerID="3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418"
Sep 16 19:24:30.887492 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.887462    2578 scope.go:117] "RemoveContainer" containerID="3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418"
Sep 16 19:24:30.887758 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:24:30.887739    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418\": container with ID starting with 3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418 not found: ID does not exist" containerID="3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418"
Sep 16 19:24:30.887807 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.887764    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418"} err="failed to get container status \"3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418\": rpc error: code = NotFound desc = could not find container \"3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418\": container with ID starting with 3d961dbf19d62918ee8963a28843fe409cbc19a15e61d0ba550a1ba81d943418 not found: ID does not exist"
Sep 16 19:24:30.896472 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.896449    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"]
Sep 16 19:24:30.901650 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.901630    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-dgmwr"]
Sep 16 19:24:30.915783 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.915762    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"]
Sep 16 19:24:30.922153 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:30.922134    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-w2zmg"]
Sep 16 19:24:31.064100 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:31.064067    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52327db1-afe1-4be9-bbf2-7dd91851308d" path="/var/lib/kubelet/pods/52327db1-afe1-4be9-bbf2-7dd91851308d/volumes"
Sep 16 19:24:31.064519 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:31.064503    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="857e7fa5-5966-438c-b980-e29b8f93cad8" path="/var/lib/kubelet/pods/857e7fa5-5966-438c-b980-e29b8f93cad8/volumes"
Sep 16 19:24:36.871351 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:36.871305    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:36.883945 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:36.883918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:36.910470 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:36.910443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:24:53.167971 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:53.167944    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:24:53.172023 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:53.172000    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:24:53.177595 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:53.177578    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:24:53.181191 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:24:53.181175    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:25:02.726573 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:02.726538    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"]
Sep 16 19:25:02.736078 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:25:02.736028    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:25:02.736078 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:02.736078 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:02.736078 ip-10-0-143-161 kubenswrapper[2578]:  > containerID="f8c4f75be6cdd056ec341dc59684aeabb05ae31fdeece00084fbe18b7bee7b5f" cmd=["/bin/sleep","15"]
Sep 16 19:25:02.736373 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:02.736101    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8" message="" err=<
Sep 16 19:25:02.736373 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:02.736373 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:02.736373 ip-10-0-143-161 kubenswrapper[2578]:  >
Sep 16 19:25:02.736373 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:25:02.736117    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:25:02.736373 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:02.736373 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:02.736373 ip-10-0-143-161 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8" podUID="17c267a1-b977-459f-b71f-ae7b8c4f3c3c" containerName="main" containerID="cri-o://f8c4f75be6cdd056ec341dc59684aeabb05ae31fdeece00084fbe18b7bee7b5f"
Sep 16 19:25:02.736373 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:02.736162    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8" podUID="17c267a1-b977-459f-b71f-ae7b8c4f3c3c" containerName="main" containerID="cri-o://f8c4f75be6cdd056ec341dc59684aeabb05ae31fdeece00084fbe18b7bee7b5f" gracePeriod=60
Sep 16 19:25:02.900707 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:02.900662    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8" podUID="17c267a1-b977-459f-b71f-ae7b8c4f3c3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 16 19:25:03.022414 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.022380    2578 generic.go:356] "Generic (PLEG): container finished" podID="17c267a1-b977-459f-b71f-ae7b8c4f3c3c" containerID="f8c4f75be6cdd056ec341dc59684aeabb05ae31fdeece00084fbe18b7bee7b5f" exitCode=0
Sep 16 19:25:03.022616 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.022452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8" event={"ID":"17c267a1-b977-459f-b71f-ae7b8c4f3c3c","Type":"ContainerDied","Data":"f8c4f75be6cdd056ec341dc59684aeabb05ae31fdeece00084fbe18b7bee7b5f"}
Sep 16 19:25:03.079751 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.079726    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:25:03.174675 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.174640    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tmp-dir\") pod \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") "
Sep 16 19:25:03.174866 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.174686    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tls-certs\") pod \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") "
Sep 16 19:25:03.174866 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.174759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-home\") pod \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") "
Sep 16 19:25:03.174866 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.174780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv2dp\" (UniqueName: \"kubernetes.io/projected/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-kube-api-access-sv2dp\") pod \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") "
Sep 16 19:25:03.174866 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.174807    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-model-cache\") pod \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") "
Sep 16 19:25:03.174866 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.174848    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-dshm\") pod \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\" (UID: \"17c267a1-b977-459f-b71f-ae7b8c4f3c3c\") "
Sep 16 19:25:03.175160 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.174892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17c267a1-b977-459f-b71f-ae7b8c4f3c3c" (UID: "17c267a1-b977-459f-b71f-ae7b8c4f3c3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:03.175160 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.174995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-home" (OuterVolumeSpecName: "home") pod "17c267a1-b977-459f-b71f-ae7b8c4f3c3c" (UID: "17c267a1-b977-459f-b71f-ae7b8c4f3c3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:03.175160 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.175097    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "17c267a1-b977-459f-b71f-ae7b8c4f3c3c" (UID: "17c267a1-b977-459f-b71f-ae7b8c4f3c3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:03.175269 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.175180    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.175269 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.175222    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.175269 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.175239    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.177133 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.177108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-dshm" (OuterVolumeSpecName: "dshm") pod "17c267a1-b977-459f-b71f-ae7b8c4f3c3c" (UID: "17c267a1-b977-459f-b71f-ae7b8c4f3c3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:03.177234 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.177141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17c267a1-b977-459f-b71f-ae7b8c4f3c3c" (UID: "17c267a1-b977-459f-b71f-ae7b8c4f3c3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:03.177234 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.177179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-kube-api-access-sv2dp" (OuterVolumeSpecName: "kube-api-access-sv2dp") pod "17c267a1-b977-459f-b71f-ae7b8c4f3c3c" (UID: "17c267a1-b977-459f-b71f-ae7b8c4f3c3c"). InnerVolumeSpecName "kube-api-access-sv2dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:03.276001 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.275913    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv2dp\" (UniqueName: \"kubernetes.io/projected/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-kube-api-access-sv2dp\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.276001 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.275945    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.276001 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:03.275954    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c267a1-b977-459f-b71f-ae7b8c4f3c3c-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:25:04.028848 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:04.028819    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"
Sep 16 19:25:04.029275 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:04.028857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8" event={"ID":"17c267a1-b977-459f-b71f-ae7b8c4f3c3c","Type":"ContainerDied","Data":"9883bef5ae48f3a0b2e066be2fcceca911986808408de4b2edfed52e486b9a67"}
Sep 16 19:25:04.029275 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:04.028912    2578 scope.go:117] "RemoveContainer" containerID="f8c4f75be6cdd056ec341dc59684aeabb05ae31fdeece00084fbe18b7bee7b5f"
Sep 16 19:25:04.052801 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:04.052772    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"]
Sep 16 19:25:04.060415 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:04.060389    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995dzjw8"]
Sep 16 19:25:05.063398 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:05.063349    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c267a1-b977-459f-b71f-ae7b8c4f3c3c" path="/var/lib/kubelet/pods/17c267a1-b977-459f-b71f-ae7b8c4f3c3c/volumes"
Sep 16 19:25:25.070381 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.070344    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"]
Sep 16 19:25:25.071194 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.071161    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c267a1-b977-459f-b71f-ae7b8c4f3c3c" containerName="main"
Sep 16 19:25:25.071194 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.071190    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c267a1-b977-459f-b71f-ae7b8c4f3c3c" containerName="main"
Sep 16 19:25:25.071389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.071206    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52327db1-afe1-4be9-bbf2-7dd91851308d" containerName="main"
Sep 16 19:25:25.071389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.071212    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52327db1-afe1-4be9-bbf2-7dd91851308d" containerName="main"
Sep 16 19:25:25.071389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.071219    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857e7fa5-5966-438c-b980-e29b8f93cad8" containerName="main"
Sep 16 19:25:25.071389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.071224    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="857e7fa5-5966-438c-b980-e29b8f93cad8" containerName="main"
Sep 16 19:25:25.071389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.071287    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c267a1-b977-459f-b71f-ae7b8c4f3c3c" containerName="main"
Sep 16 19:25:25.071389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.071298    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="857e7fa5-5966-438c-b980-e29b8f93cad8" containerName="main"
Sep 16 19:25:25.071389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.071304    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="52327db1-afe1-4be9-bbf2-7dd91851308d" containerName="main"
Sep 16 19:25:25.075951 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.075929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.078352 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.078330    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"]
Sep 16 19:25:25.079164 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.079137    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:25:25.080324 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.080303    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:25:25.080429 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.080313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 16 19:25:25.080429 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.080419    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-h7hbv\""
Sep 16 19:25:25.171472 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.171420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-dshm\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.171472 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.171467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-home\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.171749 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.171575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tls-certs\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.171749 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.171667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tmp-dir\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.171749 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.171727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppz9\" (UniqueName: \"kubernetes.io/projected/cd85cd23-3e42-452f-87ef-3e41be26cb0d-kube-api-access-2ppz9\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.171904 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.171770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-model-cache\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.273103 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.273061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-dshm\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.273103 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.273098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-home\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.273361 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.273124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tls-certs\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.273361 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.273152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tmp-dir\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.273361 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.273187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppz9\" (UniqueName: \"kubernetes.io/projected/cd85cd23-3e42-452f-87ef-3e41be26cb0d-kube-api-access-2ppz9\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.273361 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.273213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-model-cache\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.273602 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.273546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-home\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.273602 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.273579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tmp-dir\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.273704 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.273686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-model-cache\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.275508 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.275467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-dshm\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.275690 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.275670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tls-certs\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.281679 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.281659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppz9\" (UniqueName: \"kubernetes.io/projected/cd85cd23-3e42-452f-87ef-3e41be26cb0d-kube-api-access-2ppz9\") pod \"late-v2-kserve-86766546f-7x2ps\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.386908 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.386811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:25.666301 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:25.666180    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"]
Sep 16 19:25:25.669044 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:25:25.669014    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd85cd23_3e42_452f_87ef_3e41be26cb0d.slice/crio-dff12064ff8a56a33b8bf968745f8dc337466835a3ba93e443e43f2630d763ce WatchSource:0}: Error finding container dff12064ff8a56a33b8bf968745f8dc337466835a3ba93e443e43f2630d763ce: Status 404 returned error can't find the container with id dff12064ff8a56a33b8bf968745f8dc337466835a3ba93e443e43f2630d763ce
Sep 16 19:25:26.131196 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:26.131151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps" event={"ID":"cd85cd23-3e42-452f-87ef-3e41be26cb0d","Type":"ContainerStarted","Data":"0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345"}
Sep 16 19:25:26.131609 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:26.131216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps" event={"ID":"cd85cd23-3e42-452f-87ef-3e41be26cb0d","Type":"ContainerStarted","Data":"dff12064ff8a56a33b8bf968745f8dc337466835a3ba93e443e43f2630d763ce"}
Sep 16 19:25:26.152641 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:26.152590    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps" podStartSLOduration=1.152573219 podStartE2EDuration="1.152573219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:26.151080695 +0000 UTC m=+2433.598175248" watchObservedRunningTime="2026-09-16 19:25:26.152573219 +0000 UTC m=+2433.599667749"
Sep 16 19:25:26.387127 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:26.387032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:35.387928 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:35.387879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:35.400553 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:35.400524    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:25:36.188655 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:25:36.188624    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:26:06.387956 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:06.387918    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"]
Sep 16 19:26:13.471148 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.471111    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"]
Sep 16 19:26:13.474118 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.474098    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.479621 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.479596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 19:26:13.479748 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.479666    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-skjl2\""
Sep 16 19:26:13.479748 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.479683    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 19:26:13.479748 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.479596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 19:26:13.485605 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.485580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"]
Sep 16 19:26:13.601224 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.601187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-dshm\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.601392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.601242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.601392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.601267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.601392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.601292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n66t\" (UniqueName: \"kubernetes.io/projected/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-kube-api-access-2n66t\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.601392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.601333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-home\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.601392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.601348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.702404 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.702359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-dshm\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.702620 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.702457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.702620 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.702523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.702620 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.702552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n66t\" (UniqueName: \"kubernetes.io/projected/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-kube-api-access-2n66t\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.702620 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.702590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-home\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.702620 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.702607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.702902 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.702884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.702958 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.702935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.703008 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.702975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-home\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.704744 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.704721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-dshm\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.705039 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.705020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.715558 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.715535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n66t\" (UniqueName: \"kubernetes.io/projected/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-kube-api-access-2n66t\") pod \"delw-v1-kserve-6cf8cbb847-qpj6j\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:13.784663 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:13.784636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:14.058620 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:14.058593    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"]
Sep 16 19:26:14.060843 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:26:14.060817    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9a81d1_3ee6_436c_9c89_f9815f7fe24d.slice/crio-c774b4596e4c3aa10b4ab47f1f0c486f02fb16f5d2ec4c5a64a26129f94e4345 WatchSource:0}: Error finding container c774b4596e4c3aa10b4ab47f1f0c486f02fb16f5d2ec4c5a64a26129f94e4345: Status 404 returned error can't find the container with id c774b4596e4c3aa10b4ab47f1f0c486f02fb16f5d2ec4c5a64a26129f94e4345
Sep 16 19:26:14.372284 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:14.372194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j" event={"ID":"de9a81d1-3ee6-436c-9c89-f9815f7fe24d","Type":"ContainerStarted","Data":"d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224"}
Sep 16 19:26:14.372284 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:14.372245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j" event={"ID":"de9a81d1-3ee6-436c-9c89-f9815f7fe24d","Type":"ContainerStarted","Data":"c774b4596e4c3aa10b4ab47f1f0c486f02fb16f5d2ec4c5a64a26129f94e4345"}
Sep 16 19:26:14.391008 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:14.390963    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j" podStartSLOduration=1.390946713 podStartE2EDuration="1.390946713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:14.390033995 +0000 UTC m=+2481.837128544" watchObservedRunningTime="2026-09-16 19:26:14.390946713 +0000 UTC m=+2481.838041243"
Sep 16 19:26:14.785447 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:14.785416    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:21.407194 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.407156    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps" podUID="cd85cd23-3e42-452f-87ef-3e41be26cb0d" containerName="main" containerID="cri-o://0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345" gracePeriod=45
Sep 16 19:26:21.650919 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.650892    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:26:21.773912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.773818    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-home\") pod \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") "
Sep 16 19:26:21.773912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.773858    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-model-cache\") pod \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") "
Sep 16 19:26:21.773912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.773883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tls-certs\") pod \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") "
Sep 16 19:26:21.773912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.773903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ppz9\" (UniqueName: \"kubernetes.io/projected/cd85cd23-3e42-452f-87ef-3e41be26cb0d-kube-api-access-2ppz9\") pod \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") "
Sep 16 19:26:21.774252 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.773962    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-dshm\") pod \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") "
Sep 16 19:26:21.774252 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.774073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tmp-dir\") pod \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\" (UID: \"cd85cd23-3e42-452f-87ef-3e41be26cb0d\") "
Sep 16 19:26:21.774252 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.774139    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-home" (OuterVolumeSpecName: "home") pod "cd85cd23-3e42-452f-87ef-3e41be26cb0d" (UID: "cd85cd23-3e42-452f-87ef-3e41be26cb0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:21.774252 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.774153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd85cd23-3e42-452f-87ef-3e41be26cb0d" (UID: "cd85cd23-3e42-452f-87ef-3e41be26cb0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:21.774426 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.774336    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:26:21.774426 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.774357    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:26:21.774426 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.774365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd85cd23-3e42-452f-87ef-3e41be26cb0d" (UID: "cd85cd23-3e42-452f-87ef-3e41be26cb0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:21.776255 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.776230    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-dshm" (OuterVolumeSpecName: "dshm") pod "cd85cd23-3e42-452f-87ef-3e41be26cb0d" (UID: "cd85cd23-3e42-452f-87ef-3e41be26cb0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:21.776365 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.776279    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd85cd23-3e42-452f-87ef-3e41be26cb0d" (UID: "cd85cd23-3e42-452f-87ef-3e41be26cb0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:21.776365 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.776299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd85cd23-3e42-452f-87ef-3e41be26cb0d-kube-api-access-2ppz9" (OuterVolumeSpecName: "kube-api-access-2ppz9") pod "cd85cd23-3e42-452f-87ef-3e41be26cb0d" (UID: "cd85cd23-3e42-452f-87ef-3e41be26cb0d"). InnerVolumeSpecName "kube-api-access-2ppz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:21.875630 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.875594    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:26:21.875630 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.875623    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:26:21.875630 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.875633    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd85cd23-3e42-452f-87ef-3e41be26cb0d-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:26:21.875630 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:21.875641    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ppz9\" (UniqueName: \"kubernetes.io/projected/cd85cd23-3e42-452f-87ef-3e41be26cb0d-kube-api-access-2ppz9\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:26:22.410843 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:22.410800    2578 generic.go:356] "Generic (PLEG): container finished" podID="cd85cd23-3e42-452f-87ef-3e41be26cb0d" containerID="0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345" exitCode=0
Sep 16 19:26:22.411306 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:22.410858    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"
Sep 16 19:26:22.411306 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:22.410882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps" event={"ID":"cd85cd23-3e42-452f-87ef-3e41be26cb0d","Type":"ContainerDied","Data":"0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345"}
Sep 16 19:26:22.411306 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:22.410943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps" event={"ID":"cd85cd23-3e42-452f-87ef-3e41be26cb0d","Type":"ContainerDied","Data":"dff12064ff8a56a33b8bf968745f8dc337466835a3ba93e443e43f2630d763ce"}
Sep 16 19:26:22.411306 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:22.410966    2578 scope.go:117] "RemoveContainer" containerID="0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345"
Sep 16 19:26:22.419874 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:22.419855    2578 scope.go:117] "RemoveContainer" containerID="0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345"
Sep 16 19:26:22.420160 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:26:22.420137    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345\": container with ID starting with 0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345 not found: ID does not exist" containerID="0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345"
Sep 16 19:26:22.420202 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:22.420168    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345"} err="failed to get container status \"0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345\": rpc error: code = NotFound desc = could not find container \"0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345\": container with ID starting with 0c3bddeefe2569b410e65cfe1dc13c5fc95689b06c909b5744ecb6502eed1345 not found: ID does not exist"
Sep 16 19:26:22.433998 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:22.433966    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"]
Sep 16 19:26:22.440104 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:22.440079    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-7x2ps"]
Sep 16 19:26:23.064056 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:23.064022    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd85cd23-3e42-452f-87ef-3e41be26cb0d" path="/var/lib/kubelet/pods/cd85cd23-3e42-452f-87ef-3e41be26cb0d/volumes"
Sep 16 19:26:23.785282 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:23.785249    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:23.797714 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:23.797682    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:24.432201 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:24.432172    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:26:35.068539 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.068460    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"]
Sep 16 19:26:35.069046 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.069025    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd85cd23-3e42-452f-87ef-3e41be26cb0d" containerName="main"
Sep 16 19:26:35.069144 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.069050    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd85cd23-3e42-452f-87ef-3e41be26cb0d" containerName="main"
Sep 16 19:26:35.069205 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.069156    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd85cd23-3e42-452f-87ef-3e41be26cb0d" containerName="main"
Sep 16 19:26:35.072052 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.072032    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.075050 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.075032    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 19:26:35.085643 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.085625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-home\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.085728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.085651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.085728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.085686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.085728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.085723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.085837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.085747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.085837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.085801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvxs\" (UniqueName: \"kubernetes.io/projected/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-kube-api-access-wqvxs\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.124756 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.124712    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"]
Sep 16 19:26:35.186332 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.186297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvxs\" (UniqueName: \"kubernetes.io/projected/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-kube-api-access-wqvxs\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.186566 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.186344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-home\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.186566 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.186361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.186566 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.186403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.186566 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.186426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.186566 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.186444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.186808 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.186791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-home\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.186846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.186810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.186908 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.186870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.188754 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.188732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.188930 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.188911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.196131 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.196107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvxs\" (UniqueName: \"kubernetes.io/projected/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-kube-api-access-wqvxs\") pod \"delw-v2-kserve-5b84cc7ddf-bxkts\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.381786 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.381691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:35.664142 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:35.664115    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"]
Sep 16 19:26:35.666132 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:26:35.666100    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff84dd0_902e_4cd1_82aa_62a9c8880dd3.slice/crio-c0f378d4eb9712f81cbb41c188045dd9d099a5cb040b909e69f45e7f869b27d6 WatchSource:0}: Error finding container c0f378d4eb9712f81cbb41c188045dd9d099a5cb040b909e69f45e7f869b27d6: Status 404 returned error can't find the container with id c0f378d4eb9712f81cbb41c188045dd9d099a5cb040b909e69f45e7f869b27d6
Sep 16 19:26:36.482773 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:36.482739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts" event={"ID":"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3","Type":"ContainerStarted","Data":"0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e"}
Sep 16 19:26:36.483216 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:36.482799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts" event={"ID":"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3","Type":"ContainerStarted","Data":"c0f378d4eb9712f81cbb41c188045dd9d099a5cb040b909e69f45e7f869b27d6"}
Sep 16 19:26:36.501685 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:36.501635    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts" podStartSLOduration=1.501618123 podStartE2EDuration="1.501618123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:36.500072751 +0000 UTC m=+2503.947167303" watchObservedRunningTime="2026-09-16 19:26:36.501618123 +0000 UTC m=+2503.948712652"
Sep 16 19:26:37.382048 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:37.382015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:45.382607 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:45.382523    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:45.395189 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:45.395166    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:26:45.535383 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:26:45.535344    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:27:09.348431 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:09.348397    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"]
Sep 16 19:27:24.367616 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.367562    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts" podUID="0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" containerName="main" containerID="cri-o://0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e" gracePeriod=45
Sep 16 19:27:24.525152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.525111    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts" podUID="0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 16 19:27:24.615795 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.615766    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:27:24.716601 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.716509    2578 generic.go:356] "Generic (PLEG): container finished" podID="0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" containerID="0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e" exitCode=0
Sep 16 19:27:24.716601 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.716555    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"
Sep 16 19:27:24.716798 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.716597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts" event={"ID":"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3","Type":"ContainerDied","Data":"0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e"}
Sep 16 19:27:24.716798 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.716654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts" event={"ID":"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3","Type":"ContainerDied","Data":"c0f378d4eb9712f81cbb41c188045dd9d099a5cb040b909e69f45e7f869b27d6"}
Sep 16 19:27:24.716798 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.716684    2578 scope.go:117] "RemoveContainer" containerID="0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e"
Sep 16 19:27:24.722755 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.722732    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-home\") pod \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") "
Sep 16 19:27:24.722888 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.722767    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-dshm\") pod \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") "
Sep 16 19:27:24.722888 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.722803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tls-certs\") pod \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") "
Sep 16 19:27:24.722888 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.722829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-model-cache\") pod \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") "
Sep 16 19:27:24.723015 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.722915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqvxs\" (UniqueName: \"kubernetes.io/projected/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-kube-api-access-wqvxs\") pod \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") "
Sep 16 19:27:24.723015 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.722944    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tmp-dir\") pod \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\" (UID: \"0ff84dd0-902e-4cd1-82aa-62a9c8880dd3\") "
Sep 16 19:27:24.723096 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.723011    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-home" (OuterVolumeSpecName: "home") pod "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" (UID: "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:24.723209 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.723189    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:24.723209 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.723190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" (UID: "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:24.723369 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.723272    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" (UID: "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:24.725209 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.725183    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" (UID: "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:24.725325 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.725203    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-dshm" (OuterVolumeSpecName: "dshm") pod "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" (UID: "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:24.725654 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.725627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-kube-api-access-wqvxs" (OuterVolumeSpecName: "kube-api-access-wqvxs") pod "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" (UID: "0ff84dd0-902e-4cd1-82aa-62a9c8880dd3"). InnerVolumeSpecName "kube-api-access-wqvxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:24.727281 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.727260    2578 scope.go:117] "RemoveContainer" containerID="0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e"
Sep 16 19:27:24.727625 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:27:24.727602    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e\": container with ID starting with 0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e not found: ID does not exist" containerID="0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e"
Sep 16 19:27:24.727701 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.727637    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e"} err="failed to get container status \"0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e\": rpc error: code = NotFound desc = could not find container \"0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e\": container with ID starting with 0e4a9be88093b4dac6ced33d136af1d492d3fe2ce58d235908865ebfd1859f3e not found: ID does not exist"
Sep 16 19:27:24.824385 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.824351    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:24.824385 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.824378    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:24.824385 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.824389    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:24.824641 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.824418    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvxs\" (UniqueName: \"kubernetes.io/projected/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-kube-api-access-wqvxs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:24.824641 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:24.824442    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:25.040306 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:25.040275    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"]
Sep 16 19:27:25.043385 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:25.043359    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-bxkts"]
Sep 16 19:27:25.063253 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:25.063223    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" path="/var/lib/kubelet/pods/0ff84dd0-902e-4cd1-82aa-62a9c8880dd3/volumes"
Sep 16 19:27:30.351121 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.351086    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"]
Sep 16 19:27:30.351632 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.351612    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" containerName="main"
Sep 16 19:27:30.351632 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.351633    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" containerName="main"
Sep 16 19:27:30.351719 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.351696    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff84dd0-902e-4cd1-82aa-62a9c8880dd3" containerName="main"
Sep 16 19:27:30.356079 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.356057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.359708 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.359675    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-gbshk\""
Sep 16 19:27:30.360913 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.360894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:27:30.360913 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.360893    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:27:30.361068 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.360926    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:27:30.372797 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.372768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"]
Sep 16 19:27:30.471451 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.471413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.471635 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.471463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.471635 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.471538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4hds\" (UniqueName: \"kubernetes.io/projected/6577b9d0-8090-40e6-b1b7-af60090cf7b3-kube-api-access-p4hds\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.471635 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.471565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.471635 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.471591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.471635 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.471606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.572391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.572358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.572391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.572400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.572676 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.572427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.572676 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.572499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.572676 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.572552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.572676 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.572608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4hds\" (UniqueName: \"kubernetes.io/projected/6577b9d0-8090-40e6-b1b7-af60090cf7b3-kube-api-access-p4hds\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.572892 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.572862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.572962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.572920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.573019 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.572998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.575194 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.575172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.575334 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.575314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.583949 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.583925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4hds\" (UniqueName: \"kubernetes.io/projected/6577b9d0-8090-40e6-b1b7-af60090cf7b3-kube-api-access-p4hds\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.666712 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.666626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:30.957723 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:30.957671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"]
Sep 16 19:27:30.960855 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:27:30.960820    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6577b9d0_8090_40e6_b1b7_af60090cf7b3.slice/crio-282ac9712d4d1474d4048fe4d63a5fc78d77854c7191342ab4e305fd04e1869d WatchSource:0}: Error finding container 282ac9712d4d1474d4048fe4d63a5fc78d77854c7191342ab4e305fd04e1869d: Status 404 returned error can't find the container with id 282ac9712d4d1474d4048fe4d63a5fc78d77854c7191342ab4e305fd04e1869d
Sep 16 19:27:31.254547 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:31.254448    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"]
Sep 16 19:27:31.752279 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:31.752242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x" event={"ID":"6577b9d0-8090-40e6-b1b7-af60090cf7b3","Type":"ContainerStarted","Data":"9bd6b833cea32a28c4c95dfc5829da41401205ae6122ddae18b9591c21ee9e7b"}
Sep 16 19:27:31.752717 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:31.752303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x" event={"ID":"6577b9d0-8090-40e6-b1b7-af60090cf7b3","Type":"ContainerStarted","Data":"282ac9712d4d1474d4048fe4d63a5fc78d77854c7191342ab4e305fd04e1869d"}
Sep 16 19:27:31.774993 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:31.774940    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x" podStartSLOduration=1.774923641 podStartE2EDuration="1.774923641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:31.77387159 +0000 UTC m=+2559.220966151" watchObservedRunningTime="2026-09-16 19:27:31.774923641 +0000 UTC m=+2559.222018171"
Sep 16 19:27:32.667052 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:32.667016    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:40.667254 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:40.667207    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:40.679434 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:40.679409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:40.806575 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:40.806548    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:27:46.273291 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.273244    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j" podUID="de9a81d1-3ee6-436c-9c89-f9815f7fe24d" containerName="main" containerID="cri-o://d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224" gracePeriod=45
Sep 16 19:27:46.422152 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.422104    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j" podUID="de9a81d1-3ee6-436c-9c89-f9815f7fe24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 16 19:27:46.517762 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.517726    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:27:46.608010 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.607978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-dshm\") pod \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") "
Sep 16 19:27:46.608219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.608038    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n66t\" (UniqueName: \"kubernetes.io/projected/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-kube-api-access-2n66t\") pod \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") "
Sep 16 19:27:46.608219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.608084    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tmp-dir\") pod \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") "
Sep 16 19:27:46.608219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.608106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tls-certs\") pod \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") "
Sep 16 19:27:46.608219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.608135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-model-cache\") pod \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") "
Sep 16 19:27:46.608219 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.608158    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-home\") pod \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\" (UID: \"de9a81d1-3ee6-436c-9c89-f9815f7fe24d\") "
Sep 16 19:27:46.608562 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.608435    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de9a81d1-3ee6-436c-9c89-f9815f7fe24d" (UID: "de9a81d1-3ee6-436c-9c89-f9815f7fe24d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:46.608562 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.608441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-model-cache" (OuterVolumeSpecName: "model-cache") pod "de9a81d1-3ee6-436c-9c89-f9815f7fe24d" (UID: "de9a81d1-3ee6-436c-9c89-f9815f7fe24d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:46.608562 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.608509    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-home" (OuterVolumeSpecName: "home") pod "de9a81d1-3ee6-436c-9c89-f9815f7fe24d" (UID: "de9a81d1-3ee6-436c-9c89-f9815f7fe24d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:46.610362 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.610333    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de9a81d1-3ee6-436c-9c89-f9815f7fe24d" (UID: "de9a81d1-3ee6-436c-9c89-f9815f7fe24d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:46.610490 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.610374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-kube-api-access-2n66t" (OuterVolumeSpecName: "kube-api-access-2n66t") pod "de9a81d1-3ee6-436c-9c89-f9815f7fe24d" (UID: "de9a81d1-3ee6-436c-9c89-f9815f7fe24d"). InnerVolumeSpecName "kube-api-access-2n66t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:46.610490 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.610379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-dshm" (OuterVolumeSpecName: "dshm") pod "de9a81d1-3ee6-436c-9c89-f9815f7fe24d" (UID: "de9a81d1-3ee6-436c-9c89-f9815f7fe24d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:46.709509 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.709449    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n66t\" (UniqueName: \"kubernetes.io/projected/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-kube-api-access-2n66t\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:46.709509 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.709499    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:46.709509 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.709509    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:46.709509 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.709518    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:46.709782 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.709527    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:46.709782 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.709534    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de9a81d1-3ee6-436c-9c89-f9815f7fe24d-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:27:46.828797 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.828759    2578 generic.go:356] "Generic (PLEG): container finished" podID="de9a81d1-3ee6-436c-9c89-f9815f7fe24d" containerID="d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224" exitCode=0
Sep 16 19:27:46.828974 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.828802    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"
Sep 16 19:27:46.828974 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.828838    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j" event={"ID":"de9a81d1-3ee6-436c-9c89-f9815f7fe24d","Type":"ContainerDied","Data":"d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224"}
Sep 16 19:27:46.828974 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.828889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j" event={"ID":"de9a81d1-3ee6-436c-9c89-f9815f7fe24d","Type":"ContainerDied","Data":"c774b4596e4c3aa10b4ab47f1f0c486f02fb16f5d2ec4c5a64a26129f94e4345"}
Sep 16 19:27:46.828974 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.828911    2578 scope.go:117] "RemoveContainer" containerID="d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224"
Sep 16 19:27:46.837922 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.837904    2578 scope.go:117] "RemoveContainer" containerID="d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224"
Sep 16 19:27:46.838180 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:27:46.838163    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224\": container with ID starting with d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224 not found: ID does not exist" containerID="d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224"
Sep 16 19:27:46.838230 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.838190    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224"} err="failed to get container status \"d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224\": rpc error: code = NotFound desc = could not find container \"d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224\": container with ID starting with d4d0ad87243a74def27ddba94ced263d1d19ac86b5d660c38c0492ef6e48b224 not found: ID does not exist"
Sep 16 19:27:46.851534 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.851506    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"]
Sep 16 19:27:46.855704 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:46.855685    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qpj6j"]
Sep 16 19:27:47.063595 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:27:47.063554    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de9a81d1-3ee6-436c-9c89-f9815f7fe24d" path="/var/lib/kubelet/pods/de9a81d1-3ee6-436c-9c89-f9815f7fe24d/volumes"
Sep 16 19:28:23.240245 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.240165    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"]
Sep 16 19:28:23.240735 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.240639    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de9a81d1-3ee6-436c-9c89-f9815f7fe24d" containerName="main"
Sep 16 19:28:23.240735 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.240656    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de9a81d1-3ee6-436c-9c89-f9815f7fe24d" containerName="main"
Sep 16 19:28:23.240735 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.240723    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de9a81d1-3ee6-436c-9c89-f9815f7fe24d" containerName="main"
Sep 16 19:28:23.243934 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.243916    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.248707 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.248673    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 19:28:23.248707 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.248692    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-rf6cs\""
Sep 16 19:28:23.248880 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.248692    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 16 19:28:23.248880 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.248787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 19:28:23.256462 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.256438    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"]
Sep 16 19:28:23.425860 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.425821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.425860 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.425866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc327a20-79b7-4cf1-a738-5f5673a38239-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.426075 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.425889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjs9f\" (UniqueName: \"kubernetes.io/projected/bc327a20-79b7-4cf1-a738-5f5673a38239-kube-api-access-xjs9f\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.426075 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.425973    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.426075 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.426007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-home\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.426075 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.426050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-dshm\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.526998 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.526907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.526998 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.526960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc327a20-79b7-4cf1-a738-5f5673a38239-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.527193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.527005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjs9f\" (UniqueName: \"kubernetes.io/projected/bc327a20-79b7-4cf1-a738-5f5673a38239-kube-api-access-xjs9f\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.527193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.527044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.527193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.527096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-home\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.527193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.527163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-dshm\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.527381 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.527358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.527425 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.527385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.527535 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.527515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-home\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.529315 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.529295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-dshm\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.529490 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.529459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc327a20-79b7-4cf1-a738-5f5673a38239-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.536248 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.536220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjs9f\" (UniqueName: \"kubernetes.io/projected/bc327a20-79b7-4cf1-a738-5f5673a38239-kube-api-access-xjs9f\") pod \"roll-v2-kserve-d4dbf9d67-krdxc\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.555204 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.555173    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:23.827380 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:23.827351    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"]
Sep 16 19:28:23.829610 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:28:23.829579    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc327a20_79b7_4cf1_a738_5f5673a38239.slice/crio-e95669e45097cf6e4b9ceb991cbdf33e5782e03a50219e52d49472fe81032075 WatchSource:0}: Error finding container e95669e45097cf6e4b9ceb991cbdf33e5782e03a50219e52d49472fe81032075: Status 404 returned error can't find the container with id e95669e45097cf6e4b9ceb991cbdf33e5782e03a50219e52d49472fe81032075
Sep 16 19:28:24.015738 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:24.015698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc" event={"ID":"bc327a20-79b7-4cf1-a738-5f5673a38239","Type":"ContainerStarted","Data":"51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9"}
Sep 16 19:28:24.015942 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:24.015759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc" event={"ID":"bc327a20-79b7-4cf1-a738-5f5673a38239","Type":"ContainerStarted","Data":"e95669e45097cf6e4b9ceb991cbdf33e5782e03a50219e52d49472fe81032075"}
Sep 16 19:28:24.035845 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:24.035799    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc" podStartSLOduration=1.035785222 podStartE2EDuration="1.035785222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:24.034209037 +0000 UTC m=+2611.481303612" watchObservedRunningTime="2026-09-16 19:28:24.035785222 +0000 UTC m=+2611.482879753"
Sep 16 19:28:24.555881 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:24.555849    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:33.555865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:33.555818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:33.568210 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:33.568174    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:34.074834 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:34.074802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:28:55.862226 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:55.862192    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"]
Sep 16 19:28:55.872846 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:28:55.872805    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:28:55.872846 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:28:55.872846 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:28:55.872846 ip-10-0-143-161 kubenswrapper[2578]:  > containerID="9bd6b833cea32a28c4c95dfc5829da41401205ae6122ddae18b9591c21ee9e7b" cmd=["/bin/sleep","15"]
Sep 16 19:28:55.873114 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:55.872874    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x" message="" err=<
Sep 16 19:28:55.873114 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:28:55.873114 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:28:55.873114 ip-10-0-143-161 kubenswrapper[2578]:  >
Sep 16 19:28:55.873114 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:28:55.872893    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:28:55.873114 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:28:55.873114 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:28:55.873114 ip-10-0-143-161 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x" podUID="6577b9d0-8090-40e6-b1b7-af60090cf7b3" containerName="main" containerID="cri-o://9bd6b833cea32a28c4c95dfc5829da41401205ae6122ddae18b9591c21ee9e7b"
Sep 16 19:28:55.873114 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:55.872940    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x" podUID="6577b9d0-8090-40e6-b1b7-af60090cf7b3" containerName="main" containerID="cri-o://9bd6b833cea32a28c4c95dfc5829da41401205ae6122ddae18b9591c21ee9e7b" gracePeriod=60
Sep 16 19:28:56.173608 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.173578    2578 generic.go:356] "Generic (PLEG): container finished" podID="6577b9d0-8090-40e6-b1b7-af60090cf7b3" containerID="9bd6b833cea32a28c4c95dfc5829da41401205ae6122ddae18b9591c21ee9e7b" exitCode=0
Sep 16 19:28:56.173757 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.173635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x" event={"ID":"6577b9d0-8090-40e6-b1b7-af60090cf7b3","Type":"ContainerDied","Data":"9bd6b833cea32a28c4c95dfc5829da41401205ae6122ddae18b9591c21ee9e7b"}
Sep 16 19:28:56.227788 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.227760    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:28:56.321303 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.321268    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tmp-dir\") pod \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") "
Sep 16 19:28:56.321303 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.321307    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-model-cache\") pod \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") "
Sep 16 19:28:56.321571 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.321347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-dshm\") pod \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") "
Sep 16 19:28:56.321571 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.321363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-home\") pod \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") "
Sep 16 19:28:56.321571 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.321389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tls-certs\") pod \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") "
Sep 16 19:28:56.321571 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.321413    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4hds\" (UniqueName: \"kubernetes.io/projected/6577b9d0-8090-40e6-b1b7-af60090cf7b3-kube-api-access-p4hds\") pod \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\" (UID: \"6577b9d0-8090-40e6-b1b7-af60090cf7b3\") "
Sep 16 19:28:56.321759 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.321633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6577b9d0-8090-40e6-b1b7-af60090cf7b3" (UID: "6577b9d0-8090-40e6-b1b7-af60090cf7b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:56.321802 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.321786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "6577b9d0-8090-40e6-b1b7-af60090cf7b3" (UID: "6577b9d0-8090-40e6-b1b7-af60090cf7b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:56.321838 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.321812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-home" (OuterVolumeSpecName: "home") pod "6577b9d0-8090-40e6-b1b7-af60090cf7b3" (UID: "6577b9d0-8090-40e6-b1b7-af60090cf7b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:56.323770 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.323740    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-dshm" (OuterVolumeSpecName: "dshm") pod "6577b9d0-8090-40e6-b1b7-af60090cf7b3" (UID: "6577b9d0-8090-40e6-b1b7-af60090cf7b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:56.323914 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.323808    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6577b9d0-8090-40e6-b1b7-af60090cf7b3-kube-api-access-p4hds" (OuterVolumeSpecName: "kube-api-access-p4hds") pod "6577b9d0-8090-40e6-b1b7-af60090cf7b3" (UID: "6577b9d0-8090-40e6-b1b7-af60090cf7b3"). InnerVolumeSpecName "kube-api-access-p4hds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:56.323914 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.323878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6577b9d0-8090-40e6-b1b7-af60090cf7b3" (UID: "6577b9d0-8090-40e6-b1b7-af60090cf7b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:56.422669 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.422583    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:28:56.422669 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.422613    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:28:56.422669 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.422622    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:28:56.422669 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.422631    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6577b9d0-8090-40e6-b1b7-af60090cf7b3-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:28:56.422669 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.422638    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6577b9d0-8090-40e6-b1b7-af60090cf7b3-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:28:56.422669 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:56.422647    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4hds\" (UniqueName: \"kubernetes.io/projected/6577b9d0-8090-40e6-b1b7-af60090cf7b3-kube-api-access-p4hds\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:28:57.179550 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:57.179516    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"
Sep 16 19:28:57.179550 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:57.179535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x" event={"ID":"6577b9d0-8090-40e6-b1b7-af60090cf7b3","Type":"ContainerDied","Data":"282ac9712d4d1474d4048fe4d63a5fc78d77854c7191342ab4e305fd04e1869d"}
Sep 16 19:28:57.180028 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:57.179604    2578 scope.go:117] "RemoveContainer" containerID="9bd6b833cea32a28c4c95dfc5829da41401205ae6122ddae18b9591c21ee9e7b"
Sep 16 19:28:57.197688 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:57.197659    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"]
Sep 16 19:28:57.205069 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:57.205049    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-vnt4x"]
Sep 16 19:28:59.063628 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:28:59.063596    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6577b9d0-8090-40e6-b1b7-af60090cf7b3" path="/var/lib/kubelet/pods/6577b9d0-8090-40e6-b1b7-af60090cf7b3/volumes"
Sep 16 19:29:53.199329 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:29:53.199292    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:29:53.203551 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:29:53.203529    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:29:53.209598 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:29:53.209580    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:29:53.213295 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:29:53.213279    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:30:16.976920 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:16.976880    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"]
Sep 16 19:30:31.996292 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:31.996246    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc" podUID="bc327a20-79b7-4cf1-a738-5f5673a38239" containerName="main" containerID="cri-o://51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9" gracePeriod=45
Sep 16 19:30:32.064126 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.064083    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc" podUID="bc327a20-79b7-4cf1-a738-5f5673a38239" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 19:30:32.246935 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.246872    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:30:32.335179 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc327a20-79b7-4cf1-a738-5f5673a38239-tls-certs\") pod \"bc327a20-79b7-4cf1-a738-5f5673a38239\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") "
Sep 16 19:30:32.335364 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335227    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-model-cache\") pod \"bc327a20-79b7-4cf1-a738-5f5673a38239\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") "
Sep 16 19:30:32.335364 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-dshm\") pod \"bc327a20-79b7-4cf1-a738-5f5673a38239\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") "
Sep 16 19:30:32.335364 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-home\") pod \"bc327a20-79b7-4cf1-a738-5f5673a38239\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") "
Sep 16 19:30:32.335364 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjs9f\" (UniqueName: \"kubernetes.io/projected/bc327a20-79b7-4cf1-a738-5f5673a38239-kube-api-access-xjs9f\") pod \"bc327a20-79b7-4cf1-a738-5f5673a38239\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") "
Sep 16 19:30:32.335642 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-tmp-dir\") pod \"bc327a20-79b7-4cf1-a738-5f5673a38239\" (UID: \"bc327a20-79b7-4cf1-a738-5f5673a38239\") "
Sep 16 19:30:32.335642 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335568    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc327a20-79b7-4cf1-a738-5f5673a38239" (UID: "bc327a20-79b7-4cf1-a738-5f5673a38239"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:32.335763 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335689    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.335763 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335754    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc327a20-79b7-4cf1-a738-5f5673a38239" (UID: "bc327a20-79b7-4cf1-a738-5f5673a38239"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:32.336012 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.335983    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-home" (OuterVolumeSpecName: "home") pod "bc327a20-79b7-4cf1-a738-5f5673a38239" (UID: "bc327a20-79b7-4cf1-a738-5f5673a38239"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:32.337529 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.337497    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc327a20-79b7-4cf1-a738-5f5673a38239-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc327a20-79b7-4cf1-a738-5f5673a38239" (UID: "bc327a20-79b7-4cf1-a738-5f5673a38239"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:32.337622 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.337588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-dshm" (OuterVolumeSpecName: "dshm") pod "bc327a20-79b7-4cf1-a738-5f5673a38239" (UID: "bc327a20-79b7-4cf1-a738-5f5673a38239"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:32.337682 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.337634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc327a20-79b7-4cf1-a738-5f5673a38239-kube-api-access-xjs9f" (OuterVolumeSpecName: "kube-api-access-xjs9f") pod "bc327a20-79b7-4cf1-a738-5f5673a38239" (UID: "bc327a20-79b7-4cf1-a738-5f5673a38239"). InnerVolumeSpecName "kube-api-access-xjs9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:32.436828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.436790    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc327a20-79b7-4cf1-a738-5f5673a38239-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.436828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.436829    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.437042 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.436842    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.437042 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.436858    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjs9f\" (UniqueName: \"kubernetes.io/projected/bc327a20-79b7-4cf1-a738-5f5673a38239-kube-api-access-xjs9f\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.437042 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.436873    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc327a20-79b7-4cf1-a738-5f5673a38239-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:30:32.627869 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.627831    2578 generic.go:356] "Generic (PLEG): container finished" podID="bc327a20-79b7-4cf1-a738-5f5673a38239" containerID="51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9" exitCode=0
Sep 16 19:30:32.628082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.627885    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"
Sep 16 19:30:32.628082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.627916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc" event={"ID":"bc327a20-79b7-4cf1-a738-5f5673a38239","Type":"ContainerDied","Data":"51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9"}
Sep 16 19:30:32.628082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.627970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc" event={"ID":"bc327a20-79b7-4cf1-a738-5f5673a38239","Type":"ContainerDied","Data":"e95669e45097cf6e4b9ceb991cbdf33e5782e03a50219e52d49472fe81032075"}
Sep 16 19:30:32.628082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.627993    2578 scope.go:117] "RemoveContainer" containerID="51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9"
Sep 16 19:30:32.637231 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.637211    2578 scope.go:117] "RemoveContainer" containerID="51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9"
Sep 16 19:30:32.637537 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:30:32.637517    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9\": container with ID starting with 51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9 not found: ID does not exist" containerID="51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9"
Sep 16 19:30:32.637594 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.637546    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9"} err="failed to get container status \"51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9\": rpc error: code = NotFound desc = could not find container \"51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9\": container with ID starting with 51440a21dc044e951122da0c7ac21062223b12e663ce29e91083236ce0ff2ec9 not found: ID does not exist"
Sep 16 19:30:32.650473 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.650439    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"]
Sep 16 19:30:32.659414 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:32.659388    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-krdxc"]
Sep 16 19:30:33.064308 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:33.064269    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc327a20-79b7-4cf1-a738-5f5673a38239" path="/var/lib/kubelet/pods/bc327a20-79b7-4cf1-a738-5f5673a38239/volumes"
Sep 16 19:30:36.848655 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.848624    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"]
Sep 16 19:30:36.849022 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.848975    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6577b9d0-8090-40e6-b1b7-af60090cf7b3" containerName="main"
Sep 16 19:30:36.849022 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.848986    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6577b9d0-8090-40e6-b1b7-af60090cf7b3" containerName="main"
Sep 16 19:30:36.849022 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.848998    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc327a20-79b7-4cf1-a738-5f5673a38239" containerName="main"
Sep 16 19:30:36.849022 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.849004    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc327a20-79b7-4cf1-a738-5f5673a38239" containerName="main"
Sep 16 19:30:36.849158 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.849060    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6577b9d0-8090-40e6-b1b7-af60090cf7b3" containerName="main"
Sep 16 19:30:36.849158 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.849072    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc327a20-79b7-4cf1-a738-5f5673a38239" containerName="main"
Sep 16 19:30:36.852091 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.852063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:36.856779 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.856754    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 16 19:30:36.856779 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.856767    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 16 19:30:36.856962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.856770    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 16 19:30:36.856962 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.856795    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-2b8g9\""
Sep 16 19:30:36.862290 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.862263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"]
Sep 16 19:30:36.976338 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.976298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-dshm\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:36.976338 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.976339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047951e3-40c4-49ce-96b0-695d252a7980-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:36.976610 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.976359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:36.976610 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.976454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfkn\" (UniqueName: \"kubernetes.io/projected/047951e3-40c4-49ce-96b0-695d252a7980-kube-api-access-gnfkn\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:36.976610 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.976542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-home\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:36.976610 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:36.976576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-model-cache\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.077410 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.077374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfkn\" (UniqueName: \"kubernetes.io/projected/047951e3-40c4-49ce-96b0-695d252a7980-kube-api-access-gnfkn\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.077645 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.077418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-home\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.077645 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.077442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-model-cache\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.077645 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.077522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-dshm\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.077645 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.077540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047951e3-40c4-49ce-96b0-695d252a7980-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.077645 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.077560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.077892 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.077867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-home\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.077939 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.077900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.077974 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.077945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-model-cache\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.079845 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.079821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-dshm\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.080112 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.080094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047951e3-40c4-49ce-96b0-695d252a7980-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.087110 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.087084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfkn\" (UniqueName: \"kubernetes.io/projected/047951e3-40c4-49ce-96b0-695d252a7980-kube-api-access-gnfkn\") pod \"owner-v1-kserve-75579ff5bc-tjqsl\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.162787 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.162691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:37.449258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.449166    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"]
Sep 16 19:30:37.452265 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:30:37.452235    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047951e3_40c4_49ce_96b0_695d252a7980.slice/crio-3c61c279c94d1f894c14595700e11efc5360c068677a49244ee72b812e41d2d1 WatchSource:0}: Error finding container 3c61c279c94d1f894c14595700e11efc5360c068677a49244ee72b812e41d2d1: Status 404 returned error can't find the container with id 3c61c279c94d1f894c14595700e11efc5360c068677a49244ee72b812e41d2d1
Sep 16 19:30:37.660127 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.660080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl" event={"ID":"047951e3-40c4-49ce-96b0-695d252a7980","Type":"ContainerStarted","Data":"844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677"}
Sep 16 19:30:37.660298 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.660148    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl" event={"ID":"047951e3-40c4-49ce-96b0-695d252a7980","Type":"ContainerStarted","Data":"3c61c279c94d1f894c14595700e11efc5360c068677a49244ee72b812e41d2d1"}
Sep 16 19:30:37.685592 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:37.685538    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl" podStartSLOduration=1.685520662 podStartE2EDuration="1.685520662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:37.682789272 +0000 UTC m=+2745.129883823" watchObservedRunningTime="2026-09-16 19:30:37.685520662 +0000 UTC m=+2745.132615320"
Sep 16 19:30:38.163576 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:38.163535    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:47.163222 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:47.163176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:47.176127 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:47.176088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:30:47.717765 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:30:47.717735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:31:14.036297 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.036263    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"]
Sep 16 19:31:14.039733 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.039715    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.044066 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.044047    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 16 19:31:14.059888 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.059860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"]
Sep 16 19:31:14.095024 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.094987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-home\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.095024 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.095029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-model-cache\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.095286 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.095139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.095286 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.095186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-dshm\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.095286 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.095209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.095286 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.095241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pjrx\" (UniqueName: \"kubernetes.io/projected/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-kube-api-access-6pjrx\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.196034 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.195993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-home\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.196034 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.196038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-model-cache\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.196372 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.196079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.196372 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.196098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-dshm\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.196372 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.196112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.196372 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.196133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pjrx\" (UniqueName: \"kubernetes.io/projected/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-kube-api-access-6pjrx\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.196632 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.196521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-model-cache\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.196632 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.196584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-home\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.196712 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.196642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.198389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.198363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-dshm\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.198802 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.198781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.206319 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.206294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pjrx\" (UniqueName: \"kubernetes.io/projected/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-kube-api-access-6pjrx\") pod \"owner-v2-kserve-5cf8959f95-p2cbg\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.350723 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.350636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:14.625384 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.625338    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"]
Sep 16 19:31:14.627523 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:31:14.627494    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43fb869_4fda_454c_9a4c_f0165ec6bb1f.slice/crio-183d489e16b0331b1f1aaa123e9a2afadff6c6c3591fc05fcbf9572e9924a7e1 WatchSource:0}: Error finding container 183d489e16b0331b1f1aaa123e9a2afadff6c6c3591fc05fcbf9572e9924a7e1: Status 404 returned error can't find the container with id 183d489e16b0331b1f1aaa123e9a2afadff6c6c3591fc05fcbf9572e9924a7e1
Sep 16 19:31:14.829408 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.829370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg" event={"ID":"d43fb869-4fda-454c-9a4c-f0165ec6bb1f","Type":"ContainerStarted","Data":"ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d"}
Sep 16 19:31:14.829606 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.829427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg" event={"ID":"d43fb869-4fda-454c-9a4c-f0165ec6bb1f","Type":"ContainerStarted","Data":"183d489e16b0331b1f1aaa123e9a2afadff6c6c3591fc05fcbf9572e9924a7e1"}
Sep 16 19:31:14.851660 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:14.851585    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg" podStartSLOduration=1.851563708 podStartE2EDuration="1.851563708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:14.851024473 +0000 UTC m=+2782.298119028" watchObservedRunningTime="2026-09-16 19:31:14.851563708 +0000 UTC m=+2782.298658239"
Sep 16 19:31:15.351165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:15.351080    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:24.351546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:24.351510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:24.363948 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:24.363922    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:31:24.889150 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:31:24.889115    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:32:28.316059 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:28.316021    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"]
Sep 16 19:32:43.335518 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.335456    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl" podUID="047951e3-40c4-49ce-96b0-695d252a7980" containerName="main" containerID="cri-o://844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677" gracePeriod=45
Sep 16 19:32:43.584466 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.584442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:32:43.694409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694327    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-dshm\") pod \"047951e3-40c4-49ce-96b0-695d252a7980\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") "
Sep 16 19:32:43.694409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694357    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-home\") pod \"047951e3-40c4-49ce-96b0-695d252a7980\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") "
Sep 16 19:32:43.694636 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnfkn\" (UniqueName: \"kubernetes.io/projected/047951e3-40c4-49ce-96b0-695d252a7980-kube-api-access-gnfkn\") pod \"047951e3-40c4-49ce-96b0-695d252a7980\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") "
Sep 16 19:32:43.694636 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694433    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-model-cache\") pod \"047951e3-40c4-49ce-96b0-695d252a7980\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") "
Sep 16 19:32:43.694636 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694472    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-tmp-dir\") pod \"047951e3-40c4-49ce-96b0-695d252a7980\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") "
Sep 16 19:32:43.694636 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047951e3-40c4-49ce-96b0-695d252a7980-tls-certs\") pod \"047951e3-40c4-49ce-96b0-695d252a7980\" (UID: \"047951e3-40c4-49ce-96b0-695d252a7980\") "
Sep 16 19:32:43.694865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-model-cache" (OuterVolumeSpecName: "model-cache") pod "047951e3-40c4-49ce-96b0-695d252a7980" (UID: "047951e3-40c4-49ce-96b0-695d252a7980"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:43.694865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-home" (OuterVolumeSpecName: "home") pod "047951e3-40c4-49ce-96b0-695d252a7980" (UID: "047951e3-40c4-49ce-96b0-695d252a7980"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:43.694865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694833    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:32:43.694865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694850    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:32:43.694865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.694835    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "047951e3-40c4-49ce-96b0-695d252a7980" (UID: "047951e3-40c4-49ce-96b0-695d252a7980"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:43.696752 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.696723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047951e3-40c4-49ce-96b0-695d252a7980-kube-api-access-gnfkn" (OuterVolumeSpecName: "kube-api-access-gnfkn") pod "047951e3-40c4-49ce-96b0-695d252a7980" (UID: "047951e3-40c4-49ce-96b0-695d252a7980"). InnerVolumeSpecName "kube-api-access-gnfkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:43.696858 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.696750    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-dshm" (OuterVolumeSpecName: "dshm") pod "047951e3-40c4-49ce-96b0-695d252a7980" (UID: "047951e3-40c4-49ce-96b0-695d252a7980"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:43.696858 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.696731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047951e3-40c4-49ce-96b0-695d252a7980-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "047951e3-40c4-49ce-96b0-695d252a7980" (UID: "047951e3-40c4-49ce-96b0-695d252a7980"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:43.795286 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.795255    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfkn\" (UniqueName: \"kubernetes.io/projected/047951e3-40c4-49ce-96b0-695d252a7980-kube-api-access-gnfkn\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:32:43.795286 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.795279    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:32:43.795286 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.795293    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047951e3-40c4-49ce-96b0-695d252a7980-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:32:43.795612 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:43.795304    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047951e3-40c4-49ce-96b0-695d252a7980-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:32:44.252255 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:44.252211    2578 generic.go:356] "Generic (PLEG): container finished" podID="047951e3-40c4-49ce-96b0-695d252a7980" containerID="844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677" exitCode=0
Sep 16 19:32:44.252434 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:44.252264    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"
Sep 16 19:32:44.252434 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:44.252296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl" event={"ID":"047951e3-40c4-49ce-96b0-695d252a7980","Type":"ContainerDied","Data":"844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677"}
Sep 16 19:32:44.252434 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:44.252353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl" event={"ID":"047951e3-40c4-49ce-96b0-695d252a7980","Type":"ContainerDied","Data":"3c61c279c94d1f894c14595700e11efc5360c068677a49244ee72b812e41d2d1"}
Sep 16 19:32:44.252434 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:44.252383    2578 scope.go:117] "RemoveContainer" containerID="844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677"
Sep 16 19:32:44.261912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:44.261894    2578 scope.go:117] "RemoveContainer" containerID="844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677"
Sep 16 19:32:44.262193 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:32:44.262175    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677\": container with ID starting with 844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677 not found: ID does not exist" containerID="844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677"
Sep 16 19:32:44.262239 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:44.262201    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677"} err="failed to get container status \"844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677\": rpc error: code = NotFound desc = could not find container \"844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677\": container with ID starting with 844593bf2437c4eb4474e56d3c34a1eb946add1832160a67e2c9f23341553677 not found: ID does not exist"
Sep 16 19:32:44.286836 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:44.286800    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"]
Sep 16 19:32:44.290374 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:44.290350    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-tjqsl"]
Sep 16 19:32:45.064755 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:45.064720    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="047951e3-40c4-49ce-96b0-695d252a7980" path="/var/lib/kubelet/pods/047951e3-40c4-49ce-96b0-695d252a7980/volumes"
Sep 16 19:32:50.396278 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.396242    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"]
Sep 16 19:32:50.396711 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.396629    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047951e3-40c4-49ce-96b0-695d252a7980" containerName="main"
Sep 16 19:32:50.396711 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.396642    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="047951e3-40c4-49ce-96b0-695d252a7980" containerName="main"
Sep 16 19:32:50.396788 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.396718    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="047951e3-40c4-49ce-96b0-695d252a7980" containerName="main"
Sep 16 19:32:50.401361 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.401339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.405868 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.405844    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 19:32:50.405998 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.405963    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-8265v\""
Sep 16 19:32:50.406771 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.406752    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 19:32:50.407165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.407149    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:32:50.431621 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.431583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"]
Sep 16 19:32:50.550658 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.550620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fj87\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-kube-api-access-7fj87\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.550833 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.550664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-credential-socket\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.550833 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.550740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-certs\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.550833 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.550771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-socket\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.550833 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.550790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-podinfo\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.550833 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.550808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-data\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.550833 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.550826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.551039 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.550897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-token\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.551039 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.550958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-envoy\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652064 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.651975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fj87\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-kube-api-access-7fj87\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652064 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-credential-socket\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652255 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-certs\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652255 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-socket\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652255 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-podinfo\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652255 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-data\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652255 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652501 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-token\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652501 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-envoy\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652610 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-socket\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652610 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-credential-socket\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652713 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-data\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.652839 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.652819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-certs\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.653056 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.653036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.654658 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.654635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-envoy\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.654770 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.654752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-podinfo\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.661101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.661069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fj87\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-kube-api-access-7fj87\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.661281 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.661264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-token\") pod \"router-gateway-1-openshift-default-699c764795-r4rrl\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.711684 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.711642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:50.991550 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.991520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"]
Sep 16 19:32:50.993058 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:32:50.993030    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99112459_b9b1_4ae8_bfa0_5a9bc656a6c7.slice/crio-3a623d181cf923115dd63817d50a74bd62bfb57ba25d927d9e32c27f09bd319e WatchSource:0}: Error finding container 3a623d181cf923115dd63817d50a74bd62bfb57ba25d927d9e32c27f09bd319e: Status 404 returned error can't find the container with id 3a623d181cf923115dd63817d50a74bd62bfb57ba25d927d9e32c27f09bd319e
Sep 16 19:32:50.995141 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.995104    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:32:50.995245 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.995187    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:32:50.995245 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:50.995230    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:32:51.290716 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:51.290681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" event={"ID":"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7","Type":"ContainerStarted","Data":"0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e"}
Sep 16 19:32:51.290928 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:51.290739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" event={"ID":"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7","Type":"ContainerStarted","Data":"3a623d181cf923115dd63817d50a74bd62bfb57ba25d927d9e32c27f09bd319e"}
Sep 16 19:32:51.312322 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:51.312261    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" podStartSLOduration=1.312238985 podStartE2EDuration="1.312238985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:51.311724691 +0000 UTC m=+2878.758819234" watchObservedRunningTime="2026-09-16 19:32:51.312238985 +0000 UTC m=+2878.759333519"
Sep 16 19:32:51.711832 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:51.711726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:52.716827 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:52.716798    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:53.300073 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:53.300036    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:53.300949 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:53.300927    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:32:55.104389 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:32:55.104352    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"]
Sep 16 19:33:10.124192 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.124144    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg" podUID="d43fb869-4fda-454c-9a4c-f0165ec6bb1f" containerName="main" containerID="cri-o://ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d" gracePeriod=45
Sep 16 19:33:10.371456 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.371428    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:33:10.379846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.379776    2578 generic.go:356] "Generic (PLEG): container finished" podID="d43fb869-4fda-454c-9a4c-f0165ec6bb1f" containerID="ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d" exitCode=0
Sep 16 19:33:10.379846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.379830    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"
Sep 16 19:33:10.380016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.379863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg" event={"ID":"d43fb869-4fda-454c-9a4c-f0165ec6bb1f","Type":"ContainerDied","Data":"ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d"}
Sep 16 19:33:10.380016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.379914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg" event={"ID":"d43fb869-4fda-454c-9a4c-f0165ec6bb1f","Type":"ContainerDied","Data":"183d489e16b0331b1f1aaa123e9a2afadff6c6c3591fc05fcbf9572e9924a7e1"}
Sep 16 19:33:10.380016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.379938    2578 scope.go:117] "RemoveContainer" containerID="ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d"
Sep 16 19:33:10.388159 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.388142    2578 scope.go:117] "RemoveContainer" containerID="ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d"
Sep 16 19:33:10.388417 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:33:10.388397    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d\": container with ID starting with ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d not found: ID does not exist" containerID="ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d"
Sep 16 19:33:10.388512 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.388425    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d"} err="failed to get container status \"ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d\": rpc error: code = NotFound desc = could not find container \"ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d\": container with ID starting with ad5297d56f03b314cfa1ea52ebf7318f53ee671e128a5a5776707938ac832f3d not found: ID does not exist"
Sep 16 19:33:10.410596 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.410566    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pjrx\" (UniqueName: \"kubernetes.io/projected/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-kube-api-access-6pjrx\") pod \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") "
Sep 16 19:33:10.410762 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.410611    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-dshm\") pod \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") "
Sep 16 19:33:10.410762 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.410691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tls-certs\") pod \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") "
Sep 16 19:33:10.410762 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.410728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tmp-dir\") pod \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") "
Sep 16 19:33:10.410931 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.410758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-home\") pod \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") "
Sep 16 19:33:10.410931 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.410814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-model-cache\") pod \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\" (UID: \"d43fb869-4fda-454c-9a4c-f0165ec6bb1f\") "
Sep 16 19:33:10.411247 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.411216    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d43fb869-4fda-454c-9a4c-f0165ec6bb1f" (UID: "d43fb869-4fda-454c-9a4c-f0165ec6bb1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:10.411247 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.411237    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d43fb869-4fda-454c-9a4c-f0165ec6bb1f" (UID: "d43fb869-4fda-454c-9a4c-f0165ec6bb1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:10.411381 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.411225    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-home" (OuterVolumeSpecName: "home") pod "d43fb869-4fda-454c-9a4c-f0165ec6bb1f" (UID: "d43fb869-4fda-454c-9a4c-f0165ec6bb1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:10.413294 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.413262    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-dshm" (OuterVolumeSpecName: "dshm") pod "d43fb869-4fda-454c-9a4c-f0165ec6bb1f" (UID: "d43fb869-4fda-454c-9a4c-f0165ec6bb1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:10.413407 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.413297    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d43fb869-4fda-454c-9a4c-f0165ec6bb1f" (UID: "d43fb869-4fda-454c-9a4c-f0165ec6bb1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:10.413407 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.413318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-kube-api-access-6pjrx" (OuterVolumeSpecName: "kube-api-access-6pjrx") pod "d43fb869-4fda-454c-9a4c-f0165ec6bb1f" (UID: "d43fb869-4fda-454c-9a4c-f0165ec6bb1f"). InnerVolumeSpecName "kube-api-access-6pjrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:10.512238 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.512196    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pjrx\" (UniqueName: \"kubernetes.io/projected/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-kube-api-access-6pjrx\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:33:10.512238 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.512231    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:33:10.512238 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.512241    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:33:10.512510 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.512250    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:33:10.512510 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.512258    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:33:10.512510 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.512265    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fb869-4fda-454c-9a4c-f0165ec6bb1f-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:33:10.706531 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.706437    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"]
Sep 16 19:33:10.708245 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:10.708221    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-p2cbg"]
Sep 16 19:33:11.064630 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:11.064598    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d43fb869-4fda-454c-9a4c-f0165ec6bb1f" path="/var/lib/kubelet/pods/d43fb869-4fda-454c-9a4c-f0165ec6bb1f/volumes"
Sep 16 19:33:46.153291 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.153248    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"]
Sep 16 19:33:46.156613 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.154803    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43fb869-4fda-454c-9a4c-f0165ec6bb1f" containerName="main"
Sep 16 19:33:46.156613 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.154831    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43fb869-4fda-454c-9a4c-f0165ec6bb1f" containerName="main"
Sep 16 19:33:46.156613 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.155004    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43fb869-4fda-454c-9a4c-f0165ec6bb1f" containerName="main"
Sep 16 19:33:46.158810 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.158784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.162991 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.162968    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-2vq2k\""
Sep 16 19:33:46.163163 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.162972    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 16 19:33:46.164947 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.164921    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"]
Sep 16 19:33:46.232885 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.232848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.233056 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.232901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.233056 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.232956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.233056 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.233050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkw5t\" (UniqueName: \"kubernetes.io/projected/096b78ef-467c-45fc-81ef-0774f71cb0fc-kube-api-access-hkw5t\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.233200 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.233101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.233200 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.233137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096b78ef-467c-45fc-81ef-0774f71cb0fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.333817 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.333780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.333817 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.333825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.334077 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.333868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkw5t\" (UniqueName: \"kubernetes.io/projected/096b78ef-467c-45fc-81ef-0774f71cb0fc-kube-api-access-hkw5t\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.334077 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.333900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.334077 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.333920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096b78ef-467c-45fc-81ef-0774f71cb0fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.334077 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.333945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.334289 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.334220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.334341 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.334281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.334341 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.334302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.336299 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.336276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.336577 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.336560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096b78ef-467c-45fc-81ef-0774f71cb0fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.342704 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.342679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkw5t\" (UniqueName: \"kubernetes.io/projected/096b78ef-467c-45fc-81ef-0774f71cb0fc-kube-api-access-hkw5t\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.470357 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.470274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:46.744815 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:46.744725    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"]
Sep 16 19:33:46.747268 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:33:46.747229    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096b78ef_467c_45fc_81ef_0774f71cb0fc.slice/crio-b7d812cee9893956f70822995d9082de6ba17bd5bfc90afc03dfbf7cac6a8541 WatchSource:0}: Error finding container b7d812cee9893956f70822995d9082de6ba17bd5bfc90afc03dfbf7cac6a8541: Status 404 returned error can't find the container with id b7d812cee9893956f70822995d9082de6ba17bd5bfc90afc03dfbf7cac6a8541
Sep 16 19:33:47.564513 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:47.564091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz" event={"ID":"096b78ef-467c-45fc-81ef-0774f71cb0fc","Type":"ContainerStarted","Data":"3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf"}
Sep 16 19:33:47.564513 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:47.564178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz" event={"ID":"096b78ef-467c-45fc-81ef-0774f71cb0fc","Type":"ContainerStarted","Data":"b7d812cee9893956f70822995d9082de6ba17bd5bfc90afc03dfbf7cac6a8541"}
Sep 16 19:33:47.587170 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:47.587117    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz" podStartSLOduration=1.587098908 podStartE2EDuration="1.587098908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:47.584904573 +0000 UTC m=+2935.031999116" watchObservedRunningTime="2026-09-16 19:33:47.587098908 +0000 UTC m=+2935.034193438"
Sep 16 19:33:48.470711 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:48.470676    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:54.341680 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:54.341640    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:33:56.471447 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:56.471411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:56.483946 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:56.483920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:56.612583 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:56.612545    2578 generic.go:356] "Generic (PLEG): container finished" podID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerID="0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e" exitCode=0
Sep 16 19:33:56.612751 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:56.612616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" event={"ID":"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7","Type":"ContainerDied","Data":"0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e"}
Sep 16 19:33:56.613050 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:56.613031    2578 scope.go:117] "RemoveContainer" containerID="0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e"
Sep 16 19:33:56.614076 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:56.614045    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:33:56.614150 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:56.614133    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:33:56.614201 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:56.614186    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 16 19:33:56.627301 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:56.627277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:33:57.618985 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:57.618947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" event={"ID":"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7","Type":"ContainerStarted","Data":"df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3"}
Sep 16 19:33:57.712447 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:57.712408    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:33:57.712688 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:57.712653    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 19:33:58.712619 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:58.712576    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 19:33:59.712819 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:33:59.712779    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 19:34:00.712739 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:00.712700    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 19:34:01.712928 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:01.712885    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 19:34:02.712166 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:02.712124    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 19:34:03.715517 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:03.715464    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:34:03.715921 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:03.715788    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:34:03.716430 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:03.716413    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:34:53.230762 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:53.230729    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:34:53.234845 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:53.234808    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:34:53.245359 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:53.245338    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:34:53.249047 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:34:53.249029    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:38:11.379310 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.379272    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"]
Sep 16 19:38:11.389661 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:38:11.389622    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:38:11.389661 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:38:11.389661 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:38:11.389661 ip-10-0-143-161 kubenswrapper[2578]:  > containerID="3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf" cmd=["/bin/sleep","15"]
Sep 16 19:38:11.389957 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.389678    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz" message="" err=<
Sep 16 19:38:11.389957 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:38:11.389957 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:38:11.389957 ip-10-0-143-161 kubenswrapper[2578]:  >
Sep 16 19:38:11.389957 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:38:11.389690    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:38:11.389957 ip-10-0-143-161 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:38:11.389957 ip-10-0-143-161 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:38:11.389957 ip-10-0-143-161 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz" podUID="096b78ef-467c-45fc-81ef-0774f71cb0fc" containerName="main" containerID="cri-o://3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf"
Sep 16 19:38:11.389957 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.389736    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz" podUID="096b78ef-467c-45fc-81ef-0774f71cb0fc" containerName="main" containerID="cri-o://3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf" gracePeriod=60
Sep 16 19:38:11.613981 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.613944    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz" podUID="096b78ef-467c-45fc-81ef-0774f71cb0fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 16 19:38:11.735788 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.735760    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:38:11.833656 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.833621    2578 generic.go:356] "Generic (PLEG): container finished" podID="096b78ef-467c-45fc-81ef-0774f71cb0fc" containerID="3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf" exitCode=0
Sep 16 19:38:11.833830 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.833668    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"
Sep 16 19:38:11.833830 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.833710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz" event={"ID":"096b78ef-467c-45fc-81ef-0774f71cb0fc","Type":"ContainerDied","Data":"3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf"}
Sep 16 19:38:11.833830 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.833774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz" event={"ID":"096b78ef-467c-45fc-81ef-0774f71cb0fc","Type":"ContainerDied","Data":"b7d812cee9893956f70822995d9082de6ba17bd5bfc90afc03dfbf7cac6a8541"}
Sep 16 19:38:11.833830 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.833798    2578 scope.go:117] "RemoveContainer" containerID="3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf"
Sep 16 19:38:11.840780 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.840758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-home\") pod \"096b78ef-467c-45fc-81ef-0774f71cb0fc\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") "
Sep 16 19:38:11.840942 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.840803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkw5t\" (UniqueName: \"kubernetes.io/projected/096b78ef-467c-45fc-81ef-0774f71cb0fc-kube-api-access-hkw5t\") pod \"096b78ef-467c-45fc-81ef-0774f71cb0fc\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") "
Sep 16 19:38:11.840942 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.840829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-tmp-dir\") pod \"096b78ef-467c-45fc-81ef-0774f71cb0fc\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") "
Sep 16 19:38:11.840942 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.840909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-model-cache\") pod \"096b78ef-467c-45fc-81ef-0774f71cb0fc\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") "
Sep 16 19:38:11.840942 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.840934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-dshm\") pod \"096b78ef-467c-45fc-81ef-0774f71cb0fc\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") "
Sep 16 19:38:11.841176 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.840954    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096b78ef-467c-45fc-81ef-0774f71cb0fc-tls-certs\") pod \"096b78ef-467c-45fc-81ef-0774f71cb0fc\" (UID: \"096b78ef-467c-45fc-81ef-0774f71cb0fc\") "
Sep 16 19:38:11.841176 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.841058    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-home" (OuterVolumeSpecName: "home") pod "096b78ef-467c-45fc-81ef-0774f71cb0fc" (UID: "096b78ef-467c-45fc-81ef-0774f71cb0fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:11.841292 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.841216    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "096b78ef-467c-45fc-81ef-0774f71cb0fc" (UID: "096b78ef-467c-45fc-81ef-0774f71cb0fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:11.841292 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.841251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "096b78ef-467c-45fc-81ef-0774f71cb0fc" (UID: "096b78ef-467c-45fc-81ef-0774f71cb0fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:11.841292 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.841287    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:11.842885 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.842847    2578 scope.go:117] "RemoveContainer" containerID="3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf"
Sep 16 19:38:11.843151 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.843114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-dshm" (OuterVolumeSpecName: "dshm") pod "096b78ef-467c-45fc-81ef-0774f71cb0fc" (UID: "096b78ef-467c-45fc-81ef-0774f71cb0fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:11.843279 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.843175    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096b78ef-467c-45fc-81ef-0774f71cb0fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "096b78ef-467c-45fc-81ef-0774f71cb0fc" (UID: "096b78ef-467c-45fc-81ef-0774f71cb0fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:11.843279 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:38:11.843218    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf\": container with ID starting with 3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf not found: ID does not exist" containerID="3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf"
Sep 16 19:38:11.843279 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.843253    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf"} err="failed to get container status \"3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf\": rpc error: code = NotFound desc = could not find container \"3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf\": container with ID starting with 3743768b484dcb5bb695aeef44e9729625ab02c20c147edfa1d513aed20bb9cf not found: ID does not exist"
Sep 16 19:38:11.843399 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.843271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096b78ef-467c-45fc-81ef-0774f71cb0fc-kube-api-access-hkw5t" (OuterVolumeSpecName: "kube-api-access-hkw5t") pod "096b78ef-467c-45fc-81ef-0774f71cb0fc" (UID: "096b78ef-467c-45fc-81ef-0774f71cb0fc"). InnerVolumeSpecName "kube-api-access-hkw5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:11.942250 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.942152    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:11.942250 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.942191    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:11.942250 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.942199    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/096b78ef-467c-45fc-81ef-0774f71cb0fc-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:11.942250 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.942207    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096b78ef-467c-45fc-81ef-0774f71cb0fc-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:11.942250 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:11.942217    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkw5t\" (UniqueName: \"kubernetes.io/projected/096b78ef-467c-45fc-81ef-0774f71cb0fc-kube-api-access-hkw5t\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:12.157099 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:12.157062    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"]
Sep 16 19:38:12.163929 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:12.163899    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997945bz"]
Sep 16 19:38:13.063368 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:13.063329    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="096b78ef-467c-45fc-81ef-0774f71cb0fc" path="/var/lib/kubelet/pods/096b78ef-467c-45fc-81ef-0774f71cb0fc/volumes"
Sep 16 19:38:21.359799 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:21.359756    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"]
Sep 16 19:38:21.360230 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:21.359970    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy" containerID="cri-o://df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3" gracePeriod=30
Sep 16 19:38:25.479683 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.479649    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"]
Sep 16 19:38:25.480070 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.480054    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="096b78ef-467c-45fc-81ef-0774f71cb0fc" containerName="main"
Sep 16 19:38:25.480114 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.480071    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="096b78ef-467c-45fc-81ef-0774f71cb0fc" containerName="main"
Sep 16 19:38:25.480149 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.480143    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="096b78ef-467c-45fc-81ef-0774f71cb0fc" containerName="main"
Sep 16 19:38:25.483160 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.483142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.487644 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.487618    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:38:25.487801 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.487717    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 16 19:38:25.487801 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.487732    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-f2c68\""
Sep 16 19:38:25.487801 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.487737    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:38:25.487950 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.487754    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:38:25.488003 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.487990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:38:25.498891 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.498867    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"]
Sep 16 19:38:25.563934 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.563895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.564094 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.563986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.564094 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.564051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.564094 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.564075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98e2091-b9bb-4901-a9dd-3aa104937960-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.564281 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.564101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d98e2091-b9bb-4901-a9dd-3aa104937960-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.564281 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.564134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.564281 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.564179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.564281 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.564272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjj5w\" (UniqueName: \"kubernetes.io/projected/d98e2091-b9bb-4901-a9dd-3aa104937960-kube-api-access-vjj5w\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.665693 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.665654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.665693 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.665694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98e2091-b9bb-4901-a9dd-3aa104937960-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.665950 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.665716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d98e2091-b9bb-4901-a9dd-3aa104937960-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.665950 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.665735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.665950 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.665756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.665950 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.665794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjj5w\" (UniqueName: \"kubernetes.io/projected/d98e2091-b9bb-4901-a9dd-3aa104937960-kube-api-access-vjj5w\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.665950 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.665827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.665950 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.665882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.666272 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.666178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.666334 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.666307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.666392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.666362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.666492 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.666453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.666558 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.666532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d98e2091-b9bb-4901-a9dd-3aa104937960-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.668158 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.668138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.668344 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.668327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98e2091-b9bb-4901-a9dd-3aa104937960-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.676117 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.676096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjj5w\" (UniqueName: \"kubernetes.io/projected/d98e2091-b9bb-4901-a9dd-3aa104937960-kube-api-access-vjj5w\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:25.793742 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:25.793711    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:26.075032 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.074987    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"]
Sep 16 19:38:26.075786 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:38:26.075758    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98e2091_b9bb_4901_a9dd_3aa104937960.slice/crio-0c36af5fdc7b7a205f98de432440a860303fea14174e6aacf30224de80c554ef WatchSource:0}: Error finding container 0c36af5fdc7b7a205f98de432440a860303fea14174e6aacf30224de80c554ef: Status 404 returned error can't find the container with id 0c36af5fdc7b7a205f98de432440a860303fea14174e6aacf30224de80c554ef
Sep 16 19:38:26.605276 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.605252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:38:26.673405 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.673369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fj87\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-kube-api-access-7fj87\") pod \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") "
Sep 16 19:38:26.673581 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.673423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-certs\") pod \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") "
Sep 16 19:38:26.673581 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.673451    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istiod-ca-cert\") pod \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") "
Sep 16 19:38:26.673776 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.673745    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" (UID: "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:26.673904 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.673876    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" (UID: "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:38:26.674012 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.673885    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-envoy\") pod \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") "
Sep 16 19:38:26.674012 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.673991    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-podinfo\") pod \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") "
Sep 16 19:38:26.674125 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.674060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-socket\") pod \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") "
Sep 16 19:38:26.674125 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.674089    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-data\") pod \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") "
Sep 16 19:38:26.674242 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.674159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-credential-socket\") pod \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") "
Sep 16 19:38:26.674242 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.674185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-token\") pod \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\" (UID: \"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7\") "
Sep 16 19:38:26.674898 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.674454    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" (UID: "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:26.674898 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.674581    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.674898 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.674604    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istiod-ca-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.674898 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.674609    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" (UID: "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:26.674898 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.674851    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" (UID: "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:26.676193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.676158    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" (UID: "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:26.676302 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.676265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-kube-api-access-7fj87" (OuterVolumeSpecName: "kube-api-access-7fj87") pod "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" (UID: "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7"). InnerVolumeSpecName "kube-api-access-7fj87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:26.676837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.676752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" (UID: "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:38:26.677071 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.676972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" (UID: "99112459-b9b1-4ae8-bfa0-5a9bc656a6c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:26.775823 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.775780    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-workload-socket\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.775823 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.775823    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-data\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.776028 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.775837    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-credential-socket\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.776028 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.775850    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-token\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.776028 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.775863    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fj87\" (UniqueName: \"kubernetes.io/projected/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-kube-api-access-7fj87\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.776028 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.775879    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-envoy\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.776028 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.775893    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7-istio-podinfo\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.915114 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.915013    2578 generic.go:356] "Generic (PLEG): container finished" podID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerID="df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3" exitCode=0
Sep 16 19:38:26.915114 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.915076    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"
Sep 16 19:38:26.915414 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.915103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" event={"ID":"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7","Type":"ContainerDied","Data":"df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3"}
Sep 16 19:38:26.915414 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.915170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl" event={"ID":"99112459-b9b1-4ae8-bfa0-5a9bc656a6c7","Type":"ContainerDied","Data":"3a623d181cf923115dd63817d50a74bd62bfb57ba25d927d9e32c27f09bd319e"}
Sep 16 19:38:26.915414 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.915202    2578 scope.go:117] "RemoveContainer" containerID="df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3"
Sep 16 19:38:26.916978 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.916949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" event={"ID":"d98e2091-b9bb-4901-a9dd-3aa104937960","Type":"ContainerStarted","Data":"16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc"}
Sep 16 19:38:26.917104 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.917005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" event={"ID":"d98e2091-b9bb-4901-a9dd-3aa104937960","Type":"ContainerStarted","Data":"0c36af5fdc7b7a205f98de432440a860303fea14174e6aacf30224de80c554ef"}
Sep 16 19:38:26.933599 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.933572    2578 scope.go:117] "RemoveContainer" containerID="0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e"
Sep 16 19:38:26.944096 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.944072    2578 scope.go:117] "RemoveContainer" containerID="df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3"
Sep 16 19:38:26.944373 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:38:26.944353    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3\": container with ID starting with df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3 not found: ID does not exist" containerID="df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3"
Sep 16 19:38:26.944427 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.944380    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3"} err="failed to get container status \"df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3\": rpc error: code = NotFound desc = could not find container \"df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3\": container with ID starting with df89e1d34f582b41a7bd5ad887daf08a371abe5659a5ac8e116f341b67d62ca3 not found: ID does not exist"
Sep 16 19:38:26.944427 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.944397    2578 scope.go:117] "RemoveContainer" containerID="0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e"
Sep 16 19:38:26.944669 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:38:26.944653    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e\": container with ID starting with 0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e not found: ID does not exist" containerID="0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e"
Sep 16 19:38:26.944715 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.944671    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e"} err="failed to get container status \"0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e\": rpc error: code = NotFound desc = could not find container \"0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e\": container with ID starting with 0090bfe06e5fc86113018ec8858be05b148812d10f001ee4237993006544523e not found: ID does not exist"
Sep 16 19:38:26.986616 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.986581    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"]
Sep 16 19:38:26.993864 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:26.993835    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-r4rrl"]
Sep 16 19:38:27.068212 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:27.068166    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" path="/var/lib/kubelet/pods/99112459-b9b1-4ae8-bfa0-5a9bc656a6c7/volumes"
Sep 16 19:38:28.928985 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:28.928951    2578 generic.go:356] "Generic (PLEG): container finished" podID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerID="16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc" exitCode=0
Sep 16 19:38:28.929372 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:28.928997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" event={"ID":"d98e2091-b9bb-4901-a9dd-3aa104937960","Type":"ContainerDied","Data":"16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc"}
Sep 16 19:38:29.937136 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:29.937094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" event={"ID":"d98e2091-b9bb-4901-a9dd-3aa104937960","Type":"ContainerStarted","Data":"a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e"}
Sep 16 19:38:29.961470 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:29.961412    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podStartSLOduration=2.754766841 podStartE2EDuration="4.961396073s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.206629232s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:29.958422465 +0000 UTC m=+3217.405517011" watchObservedRunningTime="2026-09-16 19:38:29.961396073 +0000 UTC m=+3217.408490604"
Sep 16 19:38:30.794235 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:30.794197    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:35.794244 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:35.794207    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:38:35.795919 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:35.795891    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:38:45.794391 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:45.794290    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:38:55.794381 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:38:55.794326    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:39:05.794861 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:05.794815    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:39:15.794892 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:15.794845    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:39:25.794505 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:25.794431    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:39:35.794632 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:35.794587    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:39:45.795179 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:45.795122    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:39:53.265951 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:53.265916    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:39:53.270225 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:53.270196    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:39:53.279319 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:53.279291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:39:53.283292 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:53.283271    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:39:55.795067 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:39:55.795021    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:40:05.795171 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:05.795126    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:40:15.794823 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:15.794718    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:40:25.805285 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:25.805244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:40:25.813539 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:25.813508    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:40:33.203036 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:33.202998    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"]
Sep 16 19:40:48.234417 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:48.234366    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" containerID="cri-o://a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e" gracePeriod=45
Sep 16 19:40:48.806758 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:48.806716    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 19:40:48.980146 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:48.980113    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:40:49.047701 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.047671    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-kserve-provision-location\") pod \"d98e2091-b9bb-4901-a9dd-3aa104937960\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") "
Sep 16 19:40:49.047877 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.047711    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98e2091-b9bb-4901-a9dd-3aa104937960-tls-certs\") pod \"d98e2091-b9bb-4901-a9dd-3aa104937960\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") "
Sep 16 19:40:49.047877 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.047736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d98e2091-b9bb-4901-a9dd-3aa104937960-cabundle-cert\") pod \"d98e2091-b9bb-4901-a9dd-3aa104937960\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") "
Sep 16 19:40:49.047877 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.047790    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjj5w\" (UniqueName: \"kubernetes.io/projected/d98e2091-b9bb-4901-a9dd-3aa104937960-kube-api-access-vjj5w\") pod \"d98e2091-b9bb-4901-a9dd-3aa104937960\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") "
Sep 16 19:40:49.047877 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.047821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-tmp-dir\") pod \"d98e2091-b9bb-4901-a9dd-3aa104937960\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") "
Sep 16 19:40:49.048083 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.047886    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-home\") pod \"d98e2091-b9bb-4901-a9dd-3aa104937960\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") "
Sep 16 19:40:49.048083 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.047925    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-model-cache\") pod \"d98e2091-b9bb-4901-a9dd-3aa104937960\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") "
Sep 16 19:40:49.048083 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.047963    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-dshm\") pod \"d98e2091-b9bb-4901-a9dd-3aa104937960\" (UID: \"d98e2091-b9bb-4901-a9dd-3aa104937960\") "
Sep 16 19:40:49.048285 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.048253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98e2091-b9bb-4901-a9dd-3aa104937960-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d98e2091-b9bb-4901-a9dd-3aa104937960" (UID: "d98e2091-b9bb-4901-a9dd-3aa104937960"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:40:49.048360 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.048258    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-model-cache" (OuterVolumeSpecName: "model-cache") pod "d98e2091-b9bb-4901-a9dd-3aa104937960" (UID: "d98e2091-b9bb-4901-a9dd-3aa104937960"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:49.048719 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.048689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-home" (OuterVolumeSpecName: "home") pod "d98e2091-b9bb-4901-a9dd-3aa104937960" (UID: "d98e2091-b9bb-4901-a9dd-3aa104937960"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:49.050810 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.050776    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-dshm" (OuterVolumeSpecName: "dshm") pod "d98e2091-b9bb-4901-a9dd-3aa104937960" (UID: "d98e2091-b9bb-4901-a9dd-3aa104937960"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:49.051016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.050988    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98e2091-b9bb-4901-a9dd-3aa104937960-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d98e2091-b9bb-4901-a9dd-3aa104937960" (UID: "d98e2091-b9bb-4901-a9dd-3aa104937960"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:40:49.051116 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.051010    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98e2091-b9bb-4901-a9dd-3aa104937960-kube-api-access-vjj5w" (OuterVolumeSpecName: "kube-api-access-vjj5w") pod "d98e2091-b9bb-4901-a9dd-3aa104937960" (UID: "d98e2091-b9bb-4901-a9dd-3aa104937960"). InnerVolumeSpecName "kube-api-access-vjj5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:40:49.062281 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.062234    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d98e2091-b9bb-4901-a9dd-3aa104937960" (UID: "d98e2091-b9bb-4901-a9dd-3aa104937960"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:49.071734 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.071701    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d98e2091-b9bb-4901-a9dd-3aa104937960" (UID: "d98e2091-b9bb-4901-a9dd-3aa104937960"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:49.149448 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.149405    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjj5w\" (UniqueName: \"kubernetes.io/projected/d98e2091-b9bb-4901-a9dd-3aa104937960-kube-api-access-vjj5w\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:40:49.149448 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.149438    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:40:49.149448 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.149455    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:40:49.149750 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.149466    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:40:49.149750 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.149473    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:40:49.149750 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.149512    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d98e2091-b9bb-4901-a9dd-3aa104937960-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:40:49.149750 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.149525    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98e2091-b9bb-4901-a9dd-3aa104937960-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:40:49.149750 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.149538    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d98e2091-b9bb-4901-a9dd-3aa104937960-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:40:49.646901 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.646864    2578 generic.go:356] "Generic (PLEG): container finished" podID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerID="a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e" exitCode=0
Sep 16 19:40:49.647353 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.646913    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"
Sep 16 19:40:49.647353 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.646920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" event={"ID":"d98e2091-b9bb-4901-a9dd-3aa104937960","Type":"ContainerDied","Data":"a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e"}
Sep 16 19:40:49.647353 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.646961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw" event={"ID":"d98e2091-b9bb-4901-a9dd-3aa104937960","Type":"ContainerDied","Data":"0c36af5fdc7b7a205f98de432440a860303fea14174e6aacf30224de80c554ef"}
Sep 16 19:40:49.647353 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.646982    2578 scope.go:117] "RemoveContainer" containerID="a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e"
Sep 16 19:40:49.661351 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.661328    2578 scope.go:117] "RemoveContainer" containerID="16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc"
Sep 16 19:40:49.669709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.669686    2578 scope.go:117] "RemoveContainer" containerID="a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e"
Sep 16 19:40:49.669994 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:40:49.669975    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e\": container with ID starting with a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e not found: ID does not exist" containerID="a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e"
Sep 16 19:40:49.670040 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.670004    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e"} err="failed to get container status \"a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e\": rpc error: code = NotFound desc = could not find container \"a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e\": container with ID starting with a581942e1ab1d9b08668a5386ce697f04f11efa25da9ab4606b50fc6badec26e not found: ID does not exist"
Sep 16 19:40:49.670040 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.670022    2578 scope.go:117] "RemoveContainer" containerID="16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc"
Sep 16 19:40:49.670280 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:40:49.670261    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc\": container with ID starting with 16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc not found: ID does not exist" containerID="16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc"
Sep 16 19:40:49.670335 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.670282    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc"} err="failed to get container status \"16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc\": rpc error: code = NotFound desc = could not find container \"16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc\": container with ID starting with 16c825bbd9583b4e6e6e208303f78ccc421bfd678f4883d03c45f651d5af0dbc not found: ID does not exist"
Sep 16 19:40:49.674289 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.674268    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"]
Sep 16 19:40:49.680484 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:49.680452    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-96c995869-8bksw"]
Sep 16 19:40:51.065426 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:40:51.065393    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" path="/var/lib/kubelet/pods/d98e2091-b9bb-4901-a9dd-3aa104937960/volumes"
Sep 16 19:41:34.871793 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.871757    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"]
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872130    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872141    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872151    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="storage-initializer"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872157    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="storage-initializer"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872170    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872176    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872185    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872190    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872242    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872251    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99112459-b9b1-4ae8-bfa0-5a9bc656a6c7" containerName="istio-proxy"
Sep 16 19:41:34.872597 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.872260    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98e2091-b9bb-4901-a9dd-3aa104937960" containerName="main"
Sep 16 19:41:34.874659 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.874641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:34.877546 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.877519    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:41:34.877682 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.877545    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-mpj8t\""
Sep 16 19:41:34.877682 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.877522    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:41:34.877856 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.877839    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:41:34.886452 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.886429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"]
Sep 16 19:41:34.946348 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.946313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:34.946541 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.946354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:34.946541 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.946375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:34.946541 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.946464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:34.946668 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.946541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:34.946668 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.946577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng45j\" (UniqueName: \"kubernetes.io/projected/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kube-api-access-ng45j\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:34.946668 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:34.946601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.047673 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.047630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.047867 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.047692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.047867 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.047738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng45j\" (UniqueName: \"kubernetes.io/projected/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kube-api-access-ng45j\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.047867 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.047773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.047867 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.047810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.047867 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.047832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.047867 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.047860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.048200 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.048071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.048200 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.048131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.048287 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.048250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.048287 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.048272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.050144 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.050120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.050370 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.050349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.057293 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.057271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng45j\" (UniqueName: \"kubernetes.io/projected/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kube-api-access-ng45j\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.184937 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.184843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:35.463263 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.463051    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"]
Sep 16 19:41:35.512943 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.512921    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:41:35.865161 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:35.865119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" event={"ID":"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930","Type":"ContainerStarted","Data":"502b701f65f5e69b016c90ca797c90c9a6c52d97683128f02621647ed8b390bd"}
Sep 16 19:41:41.895686 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:41.895597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" event={"ID":"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930","Type":"ContainerStarted","Data":"07ab4bccc8984564d62ef000c4542b484264c10369bedf4c22d685143fec78c0"}
Sep 16 19:41:42.902008 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:42.901968    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerID="07ab4bccc8984564d62ef000c4542b484264c10369bedf4c22d685143fec78c0" exitCode=0
Sep 16 19:41:42.902451 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:42.902056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" event={"ID":"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930","Type":"ContainerDied","Data":"07ab4bccc8984564d62ef000c4542b484264c10369bedf4c22d685143fec78c0"}
Sep 16 19:41:43.916553 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:43.916512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" event={"ID":"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930","Type":"ContainerStarted","Data":"971c55d85b8afadb2e9718871565e13e87f2d1bd759824a5d5d9096ebe6c7c29"}
Sep 16 19:41:43.917033 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:43.916604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" event={"ID":"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930","Type":"ContainerStarted","Data":"d9c7d8719169b1b299c3153cc1e08c15f5a4c73c86e242478ace666018043e8e"}
Sep 16 19:41:43.942345 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:43.942290    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podStartSLOduration=2.67966204 podStartE2EDuration="9.942272895s" totalImagesPullingTime="6.077412275s" totalInitContainerRuntime="1.18519858s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:43.940537566 +0000 UTC m=+3411.387632124" watchObservedRunningTime="2026-09-16 19:41:43.942272895 +0000 UTC m=+3411.389367424"
Sep 16 19:41:44.185279 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:44.185244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:45.185221 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:45.185123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:41:45.186924 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:45.186890    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:41:55.185786 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:41:55.185746    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:42:05.186033 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:42:05.185987    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:42:15.185544 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:42:15.185467    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:42:25.185327 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:42:25.185280    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:42:35.185593 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:42:35.185554    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:42:45.185318 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:42:45.185269    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:42:55.185620 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:42:55.185576    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:43:05.185392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:05.185336    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:43:15.185684 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:15.185573    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:43:25.186067 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:25.186021    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:43:35.186081 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:35.186036    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:43:45.195270 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:45.195236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:43:45.203343 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:45.203315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:43:50.038718 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:50.038682    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"]
Sep 16 19:43:50.039157 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:50.038966    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="modelcar" containerID="cri-o://971c55d85b8afadb2e9718871565e13e87f2d1bd759824a5d5d9096ebe6c7c29" gracePeriod=60
Sep 16 19:43:50.594594 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:50.594565    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7_6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930/modelcar/0.log"
Sep 16 19:43:50.595245 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:50.595223    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerID="971c55d85b8afadb2e9718871565e13e87f2d1bd759824a5d5d9096ebe6c7c29" exitCode=143
Sep 16 19:43:50.595309 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:43:50.595291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" event={"ID":"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930","Type":"ContainerDied","Data":"971c55d85b8afadb2e9718871565e13e87f2d1bd759824a5d5d9096ebe6c7c29"}
Sep 16 19:44:05.065801 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:05.065757    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" containerID="cri-o://d9c7d8719169b1b299c3153cc1e08c15f5a4c73c86e242478ace666018043e8e" gracePeriod=45
Sep 16 19:44:05.195971 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:05.195931    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 19:44:05.671614 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:05.671584    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7_6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930/modelcar/0.log"
Sep 16 19:44:05.672279 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:05.672251    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerID="d9c7d8719169b1b299c3153cc1e08c15f5a4c73c86e242478ace666018043e8e" exitCode=0
Sep 16 19:44:05.672351 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:05.672279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" event={"ID":"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930","Type":"ContainerDied","Data":"d9c7d8719169b1b299c3153cc1e08c15f5a4c73c86e242478ace666018043e8e"}
Sep 16 19:44:06.318212 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.318185    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7_6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930/modelcar/0.log"
Sep 16 19:44:06.321042 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.321024    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:44:06.414527 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.414415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-home\") pod \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") "
Sep 16 19:44:06.414527 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.414469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tls-certs\") pod \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") "
Sep 16 19:44:06.414527 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.414508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kserve-provision-location\") pod \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") "
Sep 16 19:44:06.414825 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.414542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-dshm\") pod \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") "
Sep 16 19:44:06.414825 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.414577    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng45j\" (UniqueName: \"kubernetes.io/projected/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kube-api-access-ng45j\") pod \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") "
Sep 16 19:44:06.414825 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.414623    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tmp-dir\") pod \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") "
Sep 16 19:44:06.414825 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.414670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-model-cache\") pod \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\" (UID: \"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930\") "
Sep 16 19:44:06.415035 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.414926    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" (UID: "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:06.415259 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.415223    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" (UID: "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:06.415406 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.415253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-home" (OuterVolumeSpecName: "home") pod "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" (UID: "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:06.416905 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.416878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-dshm" (OuterVolumeSpecName: "dshm") pod "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" (UID: "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:06.417007 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.416947    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" (UID: "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:44:06.417007 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.416951    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kube-api-access-ng45j" (OuterVolumeSpecName: "kube-api-access-ng45j") pod "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" (UID: "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930"). InnerVolumeSpecName "kube-api-access-ng45j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:44:06.426421 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.426388    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" (UID: "6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:44:06.515819 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.515783    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:44:06.515819 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.515813    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:44:06.515819 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.515822    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:44:06.515819 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.515831    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:44:06.516090 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.515839    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:44:06.516090 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.515848    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng45j\" (UniqueName: \"kubernetes.io/projected/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-kube-api-access-ng45j\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:44:06.516090 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.515857    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:44:06.685922 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.685833    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7_6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930/modelcar/0.log"
Sep 16 19:44:06.686653 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.686622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" event={"ID":"6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930","Type":"ContainerDied","Data":"502b701f65f5e69b016c90ca797c90c9a6c52d97683128f02621647ed8b390bd"}
Sep 16 19:44:06.686754 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.686660    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"
Sep 16 19:44:06.686754 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.686729    2578 scope.go:117] "RemoveContainer" containerID="971c55d85b8afadb2e9718871565e13e87f2d1bd759824a5d5d9096ebe6c7c29"
Sep 16 19:44:06.696688 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.696670    2578 scope.go:117] "RemoveContainer" containerID="d9c7d8719169b1b299c3153cc1e08c15f5a4c73c86e242478ace666018043e8e"
Sep 16 19:44:06.706029 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.706008    2578 scope.go:117] "RemoveContainer" containerID="07ab4bccc8984564d62ef000c4542b484264c10369bedf4c22d685143fec78c0"
Sep 16 19:44:06.716922 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:44:06.716813    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1ef1aa_a991_48d7_9bcf_7dc9d34d5930.slice\": RecentStats: unable to find data in memory cache]"
Sep 16 19:44:06.718059 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.718032    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"]
Sep 16 19:44:06.721212 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:06.721190    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7"]
Sep 16 19:44:07.063737 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:07.063704    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" path="/var/lib/kubelet/pods/6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930/volumes"
Sep 16 19:44:07.196857 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:07.196814    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d6rqw7" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded"
Sep 16 19:44:51.477244 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.477210    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"]
Sep 16 19:44:51.477738 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.477718    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="modelcar-init"
Sep 16 19:44:51.477806 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.477742    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="modelcar-init"
Sep 16 19:44:51.477806 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.477756    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main"
Sep 16 19:44:51.477806 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.477765    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main"
Sep 16 19:44:51.477806 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.477778    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="modelcar"
Sep 16 19:44:51.477806 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.477785    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="modelcar"
Sep 16 19:44:51.477978 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.477857    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="modelcar"
Sep 16 19:44:51.477978 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.477865    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1ef1aa-a991-48d7-9bcf-7dc9d34d5930" containerName="main"
Sep 16 19:44:51.481006 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.480989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.484052 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.484028    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 19:44:51.484174 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.484078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 19:44:51.484362 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.484342    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 16 19:44:51.484451 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.484364    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-fv7sl\""
Sep 16 19:44:51.484787 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.484773    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:44:51.484990 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.484975    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 19:44:51.493326 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.493294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.493419 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.493359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-model-cache\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.493419 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.493382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a519bde2-b9b2-4dd4-9716-fccce21a91b0-cabundle-cert\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.493419 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.493400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tmp-dir\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.493419 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.493415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-dshm\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.493604 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.493451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrl4\" (UniqueName: \"kubernetes.io/projected/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kube-api-access-gvrl4\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.493604 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.493493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-home\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.493604 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.493529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tls-certs\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.497249 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.497227    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"]
Sep 16 19:44:51.593919 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.593841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-home\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.593919 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.593892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tls-certs\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594173 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.593948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594173 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-model-cache\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594173 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a519bde2-b9b2-4dd4-9716-fccce21a91b0-cabundle-cert\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594173 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tmp-dir\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594173 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-dshm\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594173 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrl4\" (UniqueName: \"kubernetes.io/projected/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kube-api-access-gvrl4\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594510 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-home\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594510 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-model-cache\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594510 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594780 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tmp-dir\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.594825 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.594808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a519bde2-b9b2-4dd4-9716-fccce21a91b0-cabundle-cert\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.596321 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.596303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-dshm\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.596596 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.596573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tls-certs\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.603058 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.603038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrl4\" (UniqueName: \"kubernetes.io/projected/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kube-api-access-gvrl4\") pod \"custom-route-timeout-test-kserve-768d4b4c48-95f4k\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:51.790995 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:51.790954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:44:52.078756 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:52.078727    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"]
Sep 16 19:44:52.080810 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:44:52.080786    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda519bde2_b9b2_4dd4_9716_fccce21a91b0.slice/crio-f1157f65d6d93e1cee2c23c692d2e40bad046791a322fe1f46384501f793ed1b WatchSource:0}: Error finding container f1157f65d6d93e1cee2c23c692d2e40bad046791a322fe1f46384501f793ed1b: Status 404 returned error can't find the container with id f1157f65d6d93e1cee2c23c692d2e40bad046791a322fe1f46384501f793ed1b
Sep 16 19:44:52.907556 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:52.907520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" event={"ID":"a519bde2-b9b2-4dd4-9716-fccce21a91b0","Type":"ContainerStarted","Data":"94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286"}
Sep 16 19:44:52.907951 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:52.907579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" event={"ID":"a519bde2-b9b2-4dd4-9716-fccce21a91b0","Type":"ContainerStarted","Data":"f1157f65d6d93e1cee2c23c692d2e40bad046791a322fe1f46384501f793ed1b"}
Sep 16 19:44:53.298163 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:53.298137    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:44:53.302284 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:53.302258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:44:53.313603 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:53.313577    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:44:53.324793 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:53.324760    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:44:54.918865 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:54.918825    2578 generic.go:356] "Generic (PLEG): container finished" podID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerID="94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286" exitCode=0
Sep 16 19:44:54.919360 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:54.918905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" event={"ID":"a519bde2-b9b2-4dd4-9716-fccce21a91b0","Type":"ContainerDied","Data":"94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286"}
Sep 16 19:44:55.925757 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:55.925719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" event={"ID":"a519bde2-b9b2-4dd4-9716-fccce21a91b0","Type":"ContainerStarted","Data":"67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71"}
Sep 16 19:44:55.950091 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:55.950036    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podStartSLOduration=2.757655888 podStartE2EDuration="4.950020118s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.19236423s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:44:55.947080209 +0000 UTC m=+3603.394174749" watchObservedRunningTime="2026-09-16 19:44:55.950020118 +0000 UTC m=+3603.397114647"
Sep 16 19:44:56.791285 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:44:56.791244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:45:01.791753 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:45:01.791711    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:45:01.793175 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:45:01.793150    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:45:11.792306 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:45:11.792256    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:45:21.791816 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:45:21.791766    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:45:31.792060 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:45:31.792002    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:45:41.792358 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:45:41.792299    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:45:51.791384 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:45:51.791333    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:46:01.791777 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:46:01.791731    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:46:11.791410 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:46:11.791363    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:46:21.791659 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:46:21.791559    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:46:31.791933 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:46:31.791883    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:46:41.791662 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:46:41.791615    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:46:51.801321 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:46:51.801282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:46:51.809036 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:46:51.809013    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:46:56.737736 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:46:56.737692    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"]
Sep 16 19:47:11.760471 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:11.760412    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main" containerID="cri-o://67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71" gracePeriod=45
Sep 16 19:47:12.508287 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.508259    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:47:12.577873 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.577791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvrl4\" (UniqueName: \"kubernetes.io/projected/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kube-api-access-gvrl4\") pod \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") "
Sep 16 19:47:12.577873 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.577853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a519bde2-b9b2-4dd4-9716-fccce21a91b0-cabundle-cert\") pod \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") "
Sep 16 19:47:12.578101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.577897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tmp-dir\") pod \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") "
Sep 16 19:47:12.578101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.577928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kserve-provision-location\") pod \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") "
Sep 16 19:47:12.578101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.577943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-dshm\") pod \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") "
Sep 16 19:47:12.578101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.577958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-home\") pod \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") "
Sep 16 19:47:12.578101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.577981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-model-cache\") pod \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") "
Sep 16 19:47:12.578101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.578025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tls-certs\") pod \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\" (UID: \"a519bde2-b9b2-4dd4-9716-fccce21a91b0\") "
Sep 16 19:47:12.578439 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.578367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "a519bde2-b9b2-4dd4-9716-fccce21a91b0" (UID: "a519bde2-b9b2-4dd4-9716-fccce21a91b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:12.578781 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.578745    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-home" (OuterVolumeSpecName: "home") pod "a519bde2-b9b2-4dd4-9716-fccce21a91b0" (UID: "a519bde2-b9b2-4dd4-9716-fccce21a91b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:12.578961 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.578941    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a519bde2-b9b2-4dd4-9716-fccce21a91b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a519bde2-b9b2-4dd4-9716-fccce21a91b0" (UID: "a519bde2-b9b2-4dd4-9716-fccce21a91b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:47:12.580340 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.580310    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-dshm" (OuterVolumeSpecName: "dshm") pod "a519bde2-b9b2-4dd4-9716-fccce21a91b0" (UID: "a519bde2-b9b2-4dd4-9716-fccce21a91b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:12.580526 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.580462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a519bde2-b9b2-4dd4-9716-fccce21a91b0" (UID: "a519bde2-b9b2-4dd4-9716-fccce21a91b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:47:12.580526 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.580510    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kube-api-access-gvrl4" (OuterVolumeSpecName: "kube-api-access-gvrl4") pod "a519bde2-b9b2-4dd4-9716-fccce21a91b0" (UID: "a519bde2-b9b2-4dd4-9716-fccce21a91b0"). InnerVolumeSpecName "kube-api-access-gvrl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:12.592459 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.592425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a519bde2-b9b2-4dd4-9716-fccce21a91b0" (UID: "a519bde2-b9b2-4dd4-9716-fccce21a91b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:12.603519 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.603462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a519bde2-b9b2-4dd4-9716-fccce21a91b0" (UID: "a519bde2-b9b2-4dd4-9716-fccce21a91b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:12.623713 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.623681    2578 generic.go:356] "Generic (PLEG): container finished" podID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerID="67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71" exitCode=0
Sep 16 19:47:12.623822 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.623755    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"
Sep 16 19:47:12.623822 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.623767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" event={"ID":"a519bde2-b9b2-4dd4-9716-fccce21a91b0","Type":"ContainerDied","Data":"67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71"}
Sep 16 19:47:12.623910 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.623835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k" event={"ID":"a519bde2-b9b2-4dd4-9716-fccce21a91b0","Type":"ContainerDied","Data":"f1157f65d6d93e1cee2c23c692d2e40bad046791a322fe1f46384501f793ed1b"}
Sep 16 19:47:12.623910 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.623865    2578 scope.go:117] "RemoveContainer" containerID="67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71"
Sep 16 19:47:12.633008 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.632986    2578 scope.go:117] "RemoveContainer" containerID="94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286"
Sep 16 19:47:12.642141 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.642124    2578 scope.go:117] "RemoveContainer" containerID="67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71"
Sep 16 19:47:12.643295 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:47:12.643269    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71\": container with ID starting with 67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71 not found: ID does not exist" containerID="67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71"
Sep 16 19:47:12.643405 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.643303    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71"} err="failed to get container status \"67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71\": rpc error: code = NotFound desc = could not find container \"67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71\": container with ID starting with 67de154cfabd5bbf68b266dd4365d785e5f4c4b326c7d5a194f544062d34af71 not found: ID does not exist"
Sep 16 19:47:12.643405 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.643321    2578 scope.go:117] "RemoveContainer" containerID="94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286"
Sep 16 19:47:12.643989 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:47:12.643960    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286\": container with ID starting with 94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286 not found: ID does not exist" containerID="94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286"
Sep 16 19:47:12.644067 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.643985    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286"} err="failed to get container status \"94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286\": rpc error: code = NotFound desc = could not find container \"94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286\": container with ID starting with 94a50ce19938a6915572dca84085d210dca79a4fddfbe531f7504506d0625286 not found: ID does not exist"
Sep 16 19:47:12.647319 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.647296    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"]
Sep 16 19:47:12.651300 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.651276    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-768d4b4c48-95f4k"]
Sep 16 19:47:12.678829 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.678795    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:47:12.678829 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.678828    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvrl4\" (UniqueName: \"kubernetes.io/projected/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kube-api-access-gvrl4\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:47:12.678999 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.678840    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a519bde2-b9b2-4dd4-9716-fccce21a91b0-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:47:12.678999 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.678850    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:47:12.678999 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.678860    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:47:12.678999 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.678868    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:47:12.678999 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.678877    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:47:12.678999 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:12.678885    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a519bde2-b9b2-4dd4-9716-fccce21a91b0-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:47:13.064004 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:13.063971    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" path="/var/lib/kubelet/pods/a519bde2-b9b2-4dd4-9716-fccce21a91b0/volumes"
Sep 16 19:47:52.155024 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.154977    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"]
Sep 16 19:47:52.155579 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.155513    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main"
Sep 16 19:47:52.155579 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.155534    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main"
Sep 16 19:47:52.155749 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.155585    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="storage-initializer"
Sep 16 19:47:52.155749 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.155596    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="storage-initializer"
Sep 16 19:47:52.155749 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.155680    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a519bde2-b9b2-4dd4-9716-fccce21a91b0" containerName="main"
Sep 16 19:47:52.158737 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.158716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.162503 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.162462    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 16 19:47:52.162669 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.162594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:47:52.162757 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.162674    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:47:52.163812 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.163786    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:47:52.163812 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.163807    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-rpc69\""
Sep 16 19:47:52.163992 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.163852    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:47:52.173155 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.173130    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"]
Sep 16 19:47:52.313230 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.313186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tls-certs\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.313412 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.313246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-cabundle-cert\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.313412 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.313307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-dshm\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.313412 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.313354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kserve-provision-location\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.313412 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.313395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-model-cache\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.313605 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.313422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-home\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.313605 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.313461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tmp-dir\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.313605 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.313514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwz7\" (UniqueName: \"kubernetes.io/projected/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kube-api-access-zcwz7\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.414956 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.414807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tls-certs\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.414956 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.414894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-cabundle-cert\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.414956 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.414929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-dshm\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.415246 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.414963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kserve-provision-location\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.415246 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.415030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-model-cache\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.415246 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.415069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-home\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.415246 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.415123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tmp-dir\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.415246 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.415147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwz7\" (UniqueName: \"kubernetes.io/projected/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kube-api-access-zcwz7\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.416057 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.415961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tmp-dir\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.416057 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.416015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-home\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.416234 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.416108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kserve-provision-location\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.416326 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.416300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-cabundle-cert\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.416461 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.416341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-model-cache\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.417984 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.417961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-dshm\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.418155 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.418138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tls-certs\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.429948 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.429897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwz7\" (UniqueName: \"kubernetes.io/projected/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kube-api-access-zcwz7\") pod \"router-with-refs-test-kserve-849fc9f4bd-4qgsn\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.470088 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.470052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:47:52.750911 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.750882    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"]
Sep 16 19:47:52.753055 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:47:52.753028    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeaa1181_ffa3_4b44_adc5_5b5e67f2b6ae.slice/crio-862b2c10a912fc709a06eac84db71b8abd7490d624ea0cc2067424b9b5836e28 WatchSource:0}: Error finding container 862b2c10a912fc709a06eac84db71b8abd7490d624ea0cc2067424b9b5836e28: Status 404 returned error can't find the container with id 862b2c10a912fc709a06eac84db71b8abd7490d624ea0cc2067424b9b5836e28
Sep 16 19:47:52.819321 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.819288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" event={"ID":"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae","Type":"ContainerStarted","Data":"15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c"}
Sep 16 19:47:52.819506 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:52.819347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" event={"ID":"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae","Type":"ContainerStarted","Data":"862b2c10a912fc709a06eac84db71b8abd7490d624ea0cc2067424b9b5836e28"}
Sep 16 19:47:55.834378 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:55.834340    2578 generic.go:356] "Generic (PLEG): container finished" podID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerID="15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c" exitCode=0
Sep 16 19:47:55.834878 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:55.834416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" event={"ID":"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae","Type":"ContainerDied","Data":"15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c"}
Sep 16 19:47:56.841100 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:56.841065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" event={"ID":"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae","Type":"ContainerStarted","Data":"dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6"}
Sep 16 19:47:56.865543 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:56.865470    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podStartSLOduration=2.659037905 podStartE2EDuration="4.865453387s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.206415482s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:56.863701955 +0000 UTC m=+3784.310796507" watchObservedRunningTime="2026-09-16 19:47:56.865453387 +0000 UTC m=+3784.312547925"
Sep 16 19:47:57.470564 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:47:57.470527    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:48:02.470870 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:48:02.470820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:48:02.472689 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:48:02.472650    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:48:12.470980 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:48:12.470927    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:48:22.471392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:48:22.471342    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:48:32.470851 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:48:32.470797    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:48:42.471059 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:48:42.471009    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:48:52.470788 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:48:52.470731    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:49:02.471214 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:02.471167    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:49:12.471055 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:12.471007    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:49:22.471169 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:22.471077    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:49:32.470995 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:32.470942    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:49:42.471208 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:42.471145    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 19:49:52.481329 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:52.481296    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:49:52.488992 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:52.488964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:49:53.328595 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:53.328566    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:49:53.332262 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:53.332237    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:49:53.353093 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:53.353067    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:49:53.356970 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:53.356948    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:49:57.559672 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:49:57.559636    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"]
Sep 16 19:50:12.582148 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:12.582079    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main" containerID="cri-o://dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6" gracePeriod=45
Sep 16 19:50:13.342257 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.342228    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:50:13.459151 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-home\") pod \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") "
Sep 16 19:50:13.459151 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcwz7\" (UniqueName: \"kubernetes.io/projected/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kube-api-access-zcwz7\") pod \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") "
Sep 16 19:50:13.459151 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459134    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-cabundle-cert\") pod \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") "
Sep 16 19:50:13.459416 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459188    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-dshm\") pod \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") "
Sep 16 19:50:13.459416 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-model-cache\") pod \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") "
Sep 16 19:50:13.459416 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459353    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tmp-dir\") pod \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") "
Sep 16 19:50:13.459626 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tls-certs\") pod \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") "
Sep 16 19:50:13.459626 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" (UID: "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:13.459626 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kserve-provision-location\") pod \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\" (UID: \"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae\") "
Sep 16 19:50:13.459626 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" (UID: "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:50:13.459846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459683    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-home" (OuterVolumeSpecName: "home") pod "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" (UID: "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:13.459846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459808    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:50:13.459960 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459881    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:50:13.459960 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.459898    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:50:13.462121 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.462088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-dshm" (OuterVolumeSpecName: "dshm") pod "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" (UID: "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:13.462121 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.462102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" (UID: "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:50:13.462269 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.462122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kube-api-access-zcwz7" (OuterVolumeSpecName: "kube-api-access-zcwz7") pod "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" (UID: "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae"). InnerVolumeSpecName "kube-api-access-zcwz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:13.472319 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.472278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" (UID: "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:13.480234 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.480199    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" (UID: "beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:13.530750 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.530714    2578 generic.go:356] "Generic (PLEG): container finished" podID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerID="dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6" exitCode=0
Sep 16 19:50:13.530912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.530767    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"
Sep 16 19:50:13.530912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.530772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" event={"ID":"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae","Type":"ContainerDied","Data":"dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6"}
Sep 16 19:50:13.530912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.530888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn" event={"ID":"beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae","Type":"ContainerDied","Data":"862b2c10a912fc709a06eac84db71b8abd7490d624ea0cc2067424b9b5836e28"}
Sep 16 19:50:13.531028 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.530912    2578 scope.go:117] "RemoveContainer" containerID="dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6"
Sep 16 19:50:13.541650 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.541541    2578 scope.go:117] "RemoveContainer" containerID="15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c"
Sep 16 19:50:13.550050 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.550027    2578 scope.go:117] "RemoveContainer" containerID="dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6"
Sep 16 19:50:13.550345 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:50:13.550324    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6\": container with ID starting with dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6 not found: ID does not exist" containerID="dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6"
Sep 16 19:50:13.550385 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.550355    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6"} err="failed to get container status \"dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6\": rpc error: code = NotFound desc = could not find container \"dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6\": container with ID starting with dd845d2aebea6dc025260820c770fca7c533355ab4aea49cad02cdd0e2431aa6 not found: ID does not exist"
Sep 16 19:50:13.550385 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.550375    2578 scope.go:117] "RemoveContainer" containerID="15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c"
Sep 16 19:50:13.550655 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:50:13.550639    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c\": container with ID starting with 15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c not found: ID does not exist" containerID="15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c"
Sep 16 19:50:13.550701 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.550661    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c"} err="failed to get container status \"15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c\": rpc error: code = NotFound desc = could not find container \"15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c\": container with ID starting with 15d91ad2bd3346fadc1a8009e08cdb4f80d40bd4d911cc356b7b8c9d2387000c not found: ID does not exist"
Sep 16 19:50:13.554679 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.554656    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"]
Sep 16 19:50:13.558029 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.558008    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-849fc9f4bd-4qgsn"]
Sep 16 19:50:13.560585 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.560565    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcwz7\" (UniqueName: \"kubernetes.io/projected/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kube-api-access-zcwz7\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:50:13.560661 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.560587    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:50:13.560661 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.560597    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:50:13.560661 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.560607    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:50:13.560661 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:13.560615    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:50:15.063745 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:15.063707    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" path="/var/lib/kubelet/pods/beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae/volumes"
Sep 16 19:50:46.170598 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.170510    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"]
Sep 16 19:50:46.171065 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.170899    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="storage-initializer"
Sep 16 19:50:46.171065 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.170915    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="storage-initializer"
Sep 16 19:50:46.171065 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.170928    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main"
Sep 16 19:50:46.171065 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.170934    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main"
Sep 16 19:50:46.171065 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.171006    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="beaa1181-ffa3-4b44-adc5-5b5e67f2b6ae" containerName="main"
Sep 16 19:50:46.173912 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.173896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:46.176757 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.176736    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 16 19:50:46.176927 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.176908    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 19:50:46.177909 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.177886    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-hd8vj\""
Sep 16 19:50:46.178001 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.177915    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 19:50:46.182135 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.182107    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"]
Sep 16 19:50:46.346031 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.345989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zf7l\" (UniqueName: \"kubernetes.io/projected/0d02bed1-327e-4bc1-9a83-3427c9884465-kube-api-access-2zf7l\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:46.447407 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.447313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zf7l\" (UniqueName: \"kubernetes.io/projected/0d02bed1-327e-4bc1-9a83-3427c9884465-kube-api-access-2zf7l\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:46.457589 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:46.457557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zf7l\" (UniqueName: \"kubernetes.io/projected/0d02bed1-327e-4bc1-9a83-3427c9884465-kube-api-access-2zf7l\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:49.169399 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:49.169354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:49.176354 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:49.176325    2578 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:49.269923 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:49.269879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:49.270130 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:49.270021    2578 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:49.273159 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:49.273135    2578 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") DevicePath \"csi-ec5c1ce12fb9524044da6dd93b25eb1de0547c362573d34bb2aeea84e15dfa8f\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:49.502694 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:49.502657    2578 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/e3bf9cefc8fe9ee845aae35a5e89c33b4256cbb38db14601f2e77b7858f8a858/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:49.517830 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:49.517795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"e2e-pvc-lora-download-fwgcd\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:49.783516 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:49.783459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:50.056654 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:50.056586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"]
Sep 16 19:50:50.059012 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:50:50.058983    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d02bed1_327e_4bc1_9a83_3427c9884465.slice/crio-e32e7ee7a4d3df77784c57bb9ba3c84d8b71aaef7d42f1bdeb3550e55b9b8fee WatchSource:0}: Error finding container e32e7ee7a4d3df77784c57bb9ba3c84d8b71aaef7d42f1bdeb3550e55b9b8fee: Status 404 returned error can't find the container with id e32e7ee7a4d3df77784c57bb9ba3c84d8b71aaef7d42f1bdeb3550e55b9b8fee
Sep 16 19:50:50.715801 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:50.715761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd" event={"ID":"0d02bed1-327e-4bc1-9a83-3427c9884465","Type":"ContainerStarted","Data":"abdf062a73db2c07c75f221c70977541f2cad1868337a2966038edd36137dd3b"}
Sep 16 19:50:50.716199 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:50.715814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd" event={"ID":"0d02bed1-327e-4bc1-9a83-3427c9884465","Type":"ContainerStarted","Data":"e32e7ee7a4d3df77784c57bb9ba3c84d8b71aaef7d42f1bdeb3550e55b9b8fee"}
Sep 16 19:50:50.741258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:50.741203    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd" podStartSLOduration=7.741186595 podStartE2EDuration="7.741186595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:50.738317186 +0000 UTC m=+3958.185411716" watchObservedRunningTime="2026-09-16 19:50:50.741186595 +0000 UTC m=+3958.188281134"
Sep 16 19:50:52.726113 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:52.726078    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d02bed1-327e-4bc1-9a83-3427c9884465" containerID="abdf062a73db2c07c75f221c70977541f2cad1868337a2966038edd36137dd3b" exitCode=0
Sep 16 19:50:52.726567 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:52.726153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd" event={"ID":"0d02bed1-327e-4bc1-9a83-3427c9884465","Type":"ContainerDied","Data":"abdf062a73db2c07c75f221c70977541f2cad1868337a2966038edd36137dd3b"}
Sep 16 19:50:53.875576 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:53.875542    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:54.011670 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:54.011569    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zf7l\" (UniqueName: \"kubernetes.io/projected/0d02bed1-327e-4bc1-9a83-3427c9884465-kube-api-access-2zf7l\") pod \"0d02bed1-327e-4bc1-9a83-3427c9884465\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") "
Sep 16 19:50:54.011833 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:54.011745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") pod \"0d02bed1-327e-4bc1-9a83-3427c9884465\" (UID: \"0d02bed1-327e-4bc1-9a83-3427c9884465\") "
Sep 16 19:50:54.014043 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:54.014008    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d02bed1-327e-4bc1-9a83-3427c9884465-kube-api-access-2zf7l" (OuterVolumeSpecName: "kube-api-access-2zf7l") pod "0d02bed1-327e-4bc1-9a83-3427c9884465" (UID: "0d02bed1-327e-4bc1-9a83-3427c9884465"). InnerVolumeSpecName "kube-api-access-2zf7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:54.016070 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:54.016044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99" (OuterVolumeSpecName: "model-storage") pod "0d02bed1-327e-4bc1-9a83-3427c9884465" (UID: "0d02bed1-327e-4bc1-9a83-3427c9884465"). InnerVolumeSpecName "pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:50:54.112412 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:54.112370    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zf7l\" (UniqueName: \"kubernetes.io/projected/0d02bed1-327e-4bc1-9a83-3427c9884465-kube-api-access-2zf7l\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:50:54.112612 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:54.112425    2578 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") on node \"ip-10-0-143-161.ec2.internal\" "
Sep 16 19:50:54.739235 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:54.739206    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"
Sep 16 19:50:54.739407 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:54.739246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd" event={"ID":"0d02bed1-327e-4bc1-9a83-3427c9884465","Type":"ContainerDied","Data":"e32e7ee7a4d3df77784c57bb9ba3c84d8b71aaef7d42f1bdeb3550e55b9b8fee"}
Sep 16 19:50:54.739407 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:54.739283    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e32e7ee7a4d3df77784c57bb9ba3c84d8b71aaef7d42f1bdeb3550e55b9b8fee"
Sep 16 19:50:55.085987 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:55.085962    2578 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99") on node "ip-10-0-143-161.ec2.internal"
Sep 16 19:50:55.121046 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:50:55.121008    2578 reconciler_common.go:299] "Volume detached for volume \"pvc-3e2380d5-da3d-47fe-9623-4f79b19446d4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09e866a5a80566f99\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"csi-ec5c1ce12fb9524044da6dd93b25eb1de0547c362573d34bb2aeea84e15dfa8f\""
Sep 16 19:51:10.248713 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.248673    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"]
Sep 16 19:51:10.249313 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.249159    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d02bed1-327e-4bc1-9a83-3427c9884465" containerName="storage-initializer"
Sep 16 19:51:10.249313 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.249177    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d02bed1-327e-4bc1-9a83-3427c9884465" containerName="storage-initializer"
Sep 16 19:51:10.249313 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.249255    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d02bed1-327e-4bc1-9a83-3427c9884465" containerName="storage-initializer"
Sep 16 19:51:10.253709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.253685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.257396 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.257370    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:51:10.257582 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.257429    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:51:10.258330 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.258307    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:51:10.258330 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.258327    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:51:10.258523 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.258307    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:51:10.258523 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.258382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-mkkwr\""
Sep 16 19:51:10.265948 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.265923    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"]
Sep 16 19:51:10.338955 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.338912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.338955 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.338956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.339242 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.339033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a33671-8fa6-4687-8d12-3b400364039f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.339242 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.339132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33a33671-8fa6-4687-8d12-3b400364039f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.339242 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.339159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.339242 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.339184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.339404 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.339245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cs6w\" (UniqueName: \"kubernetes.io/projected/33a33671-8fa6-4687-8d12-3b400364039f-kube-api-access-6cs6w\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.339404 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.339295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440386 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33a33671-8fa6-4687-8d12-3b400364039f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440386 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440687 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440687 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cs6w\" (UniqueName: \"kubernetes.io/projected/33a33671-8fa6-4687-8d12-3b400364039f-kube-api-access-6cs6w\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440687 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440687 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440687 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440687 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a33671-8fa6-4687-8d12-3b400364039f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440994 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440832    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440994 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.440994 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.441160 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.440999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.441404 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.441382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33a33671-8fa6-4687-8d12-3b400364039f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.442899 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.442880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.443204 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.443186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a33671-8fa6-4687-8d12-3b400364039f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.449279 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.449254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cs6w\" (UniqueName: \"kubernetes.io/projected/33a33671-8fa6-4687-8d12-3b400364039f-kube-api-access-6cs6w\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.565836 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.565792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:10.855603 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:10.855504    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"]
Sep 16 19:51:10.857615 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:51:10.857587    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a33671_8fa6_4687_8d12_3b400364039f.slice/crio-ba81ce52b52356afe9095ca97696e2d1fc40782ee07f280a198635d38e5f3f33 WatchSource:0}: Error finding container ba81ce52b52356afe9095ca97696e2d1fc40782ee07f280a198635d38e5f3f33: Status 404 returned error can't find the container with id ba81ce52b52356afe9095ca97696e2d1fc40782ee07f280a198635d38e5f3f33
Sep 16 19:51:11.825016 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:11.824974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" event={"ID":"33a33671-8fa6-4687-8d12-3b400364039f","Type":"ContainerStarted","Data":"6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6"}
Sep 16 19:51:11.825444 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:11.825030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" event={"ID":"33a33671-8fa6-4687-8d12-3b400364039f","Type":"ContainerStarted","Data":"ba81ce52b52356afe9095ca97696e2d1fc40782ee07f280a198635d38e5f3f33"}
Sep 16 19:51:13.835661 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:13.835623    2578 generic.go:356] "Generic (PLEG): container finished" podID="33a33671-8fa6-4687-8d12-3b400364039f" containerID="6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6" exitCode=0
Sep 16 19:51:13.836051 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:13.835697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" event={"ID":"33a33671-8fa6-4687-8d12-3b400364039f","Type":"ContainerDied","Data":"6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6"}
Sep 16 19:51:14.842772 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:14.842726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" event={"ID":"33a33671-8fa6-4687-8d12-3b400364039f","Type":"ContainerStarted","Data":"4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844"}
Sep 16 19:51:14.872356 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:14.872299    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podStartSLOduration=2.643287274 podStartE2EDuration="4.872281411s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.228994137s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:14.871186081 +0000 UTC m=+3982.318280637" watchObservedRunningTime="2026-09-16 19:51:14.872281411 +0000 UTC m=+3982.319375941"
Sep 16 19:51:20.565914 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:20.565874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:20.566577 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:20.565949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:51:20.567585 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:20.567547    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:51:30.566633 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:30.566580    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:51:40.566608 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:40.566545    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:51:50.566415 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:51:50.566362    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:52:00.566759 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:52:00.566711    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:52:10.566544 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:52:10.566465    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:52:20.567025 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:52:20.566924    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:52:30.566814 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:52:30.566766    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:52:40.566573 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:52:40.566519    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:52:50.567515 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:52:50.567459    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:53:00.567685 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:00.567631    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 19:53:10.576464 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:10.576425    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:53:10.585071 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:10.585038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:53:20.156056 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:20.156015    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"]
Sep 16 19:53:35.178928 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.178882    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main" containerID="cri-o://4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844" gracePeriod=45
Sep 16 19:53:35.925989 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.925953    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:53:35.964939 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.964857    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-dshm\") pod \"33a33671-8fa6-4687-8d12-3b400364039f\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") "
Sep 16 19:53:35.964939 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.964894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a33671-8fa6-4687-8d12-3b400364039f-tls-certs\") pod \"33a33671-8fa6-4687-8d12-3b400364039f\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") "
Sep 16 19:53:35.965199 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.964956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-kserve-provision-location\") pod \"33a33671-8fa6-4687-8d12-3b400364039f\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") "
Sep 16 19:53:35.965199 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.964980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-home\") pod \"33a33671-8fa6-4687-8d12-3b400364039f\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") "
Sep 16 19:53:35.965199 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.965014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cs6w\" (UniqueName: \"kubernetes.io/projected/33a33671-8fa6-4687-8d12-3b400364039f-kube-api-access-6cs6w\") pod \"33a33671-8fa6-4687-8d12-3b400364039f\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") "
Sep 16 19:53:35.965199 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.965044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33a33671-8fa6-4687-8d12-3b400364039f-cabundle-cert\") pod \"33a33671-8fa6-4687-8d12-3b400364039f\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") "
Sep 16 19:53:35.965199 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.965088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-model-cache\") pod \"33a33671-8fa6-4687-8d12-3b400364039f\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") "
Sep 16 19:53:35.965199 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.965116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-tmp-dir\") pod \"33a33671-8fa6-4687-8d12-3b400364039f\" (UID: \"33a33671-8fa6-4687-8d12-3b400364039f\") "
Sep 16 19:53:35.965551 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.965513    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-model-cache" (OuterVolumeSpecName: "model-cache") pod "33a33671-8fa6-4687-8d12-3b400364039f" (UID: "33a33671-8fa6-4687-8d12-3b400364039f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:35.965746 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.965724    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a33671-8fa6-4687-8d12-3b400364039f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33a33671-8fa6-4687-8d12-3b400364039f" (UID: "33a33671-8fa6-4687-8d12-3b400364039f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:53:35.965822 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.965796    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-home" (OuterVolumeSpecName: "home") pod "33a33671-8fa6-4687-8d12-3b400364039f" (UID: "33a33671-8fa6-4687-8d12-3b400364039f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:35.967968 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.967927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-dshm" (OuterVolumeSpecName: "dshm") pod "33a33671-8fa6-4687-8d12-3b400364039f" (UID: "33a33671-8fa6-4687-8d12-3b400364039f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:35.967968 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.967936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a33671-8fa6-4687-8d12-3b400364039f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33a33671-8fa6-4687-8d12-3b400364039f" (UID: "33a33671-8fa6-4687-8d12-3b400364039f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:53:35.967968 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.967947    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a33671-8fa6-4687-8d12-3b400364039f-kube-api-access-6cs6w" (OuterVolumeSpecName: "kube-api-access-6cs6w") pod "33a33671-8fa6-4687-8d12-3b400364039f" (UID: "33a33671-8fa6-4687-8d12-3b400364039f"). InnerVolumeSpecName "kube-api-access-6cs6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:53:35.978622 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.978593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33a33671-8fa6-4687-8d12-3b400364039f" (UID: "33a33671-8fa6-4687-8d12-3b400364039f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:35.989527 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:35.989462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33a33671-8fa6-4687-8d12-3b400364039f" (UID: "33a33671-8fa6-4687-8d12-3b400364039f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:36.066434 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.066399    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.066434 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.066431    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.066434 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.066442    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cs6w\" (UniqueName: \"kubernetes.io/projected/33a33671-8fa6-4687-8d12-3b400364039f-kube-api-access-6cs6w\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.066709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.066452    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33a33671-8fa6-4687-8d12-3b400364039f-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.066709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.066462    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.066709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.066469    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.066709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.066500    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a33671-8fa6-4687-8d12-3b400364039f-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.066709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.066509    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a33671-8fa6-4687-8d12-3b400364039f-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.559240 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.559202    2578 generic.go:356] "Generic (PLEG): container finished" podID="33a33671-8fa6-4687-8d12-3b400364039f" containerID="4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844" exitCode=0
Sep 16 19:53:36.559731 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.559253    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"
Sep 16 19:53:36.559731 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.559278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" event={"ID":"33a33671-8fa6-4687-8d12-3b400364039f","Type":"ContainerDied","Data":"4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844"}
Sep 16 19:53:36.559731 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.559327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt" event={"ID":"33a33671-8fa6-4687-8d12-3b400364039f","Type":"ContainerDied","Data":"ba81ce52b52356afe9095ca97696e2d1fc40782ee07f280a198635d38e5f3f33"}
Sep 16 19:53:36.559731 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.559351    2578 scope.go:117] "RemoveContainer" containerID="4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844"
Sep 16 19:53:36.569345 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.569327    2578 scope.go:117] "RemoveContainer" containerID="6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6"
Sep 16 19:53:36.577464 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.577442    2578 scope.go:117] "RemoveContainer" containerID="4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844"
Sep 16 19:53:36.577766 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:53:36.577745    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844\": container with ID starting with 4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844 not found: ID does not exist" containerID="4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844"
Sep 16 19:53:36.577818 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.577776    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844"} err="failed to get container status \"4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844\": rpc error: code = NotFound desc = could not find container \"4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844\": container with ID starting with 4df83b7cc59509dd16e7dee02713d24dcbb5c51c67b6a858b320f66624776844 not found: ID does not exist"
Sep 16 19:53:36.577818 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.577795    2578 scope.go:117] "RemoveContainer" containerID="6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6"
Sep 16 19:53:36.578022 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:53:36.578002    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6\": container with ID starting with 6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6 not found: ID does not exist" containerID="6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6"
Sep 16 19:53:36.578073 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.578029    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6"} err="failed to get container status \"6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6\": rpc error: code = NotFound desc = could not find container \"6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6\": container with ID starting with 6b39e715b3bd8b2e96d3705d125481dc4ffa2b31c78e6b15cfb2fa6dee52daf6 not found: ID does not exist"
Sep 16 19:53:36.585412 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.585385    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"]
Sep 16 19:53:36.590365 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:36.590342    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-76jk5bt"]
Sep 16 19:53:37.063792 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:53:37.063756    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a33671-8fa6-4687-8d12-3b400364039f" path="/var/lib/kubelet/pods/33a33671-8fa6-4687-8d12-3b400364039f/volumes"
Sep 16 19:54:05.210371 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:05.210331    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"]
Sep 16 19:54:05.213847 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:05.213813    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-fwgcd"]
Sep 16 19:54:07.064449 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:07.064410    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d02bed1-327e-4bc1-9a83-3427c9884465" path="/var/lib/kubelet/pods/0d02bed1-327e-4bc1-9a83-3427c9884465/volumes"
Sep 16 19:54:11.681886 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.681844    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"]
Sep 16 19:54:11.682430 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.682408    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="storage-initializer"
Sep 16 19:54:11.682509 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.682432    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="storage-initializer"
Sep 16 19:54:11.682509 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.682442    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main"
Sep 16 19:54:11.682509 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.682448    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main"
Sep 16 19:54:11.682627 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.682533    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a33671-8fa6-4687-8d12-3b400364039f" containerName="main"
Sep 16 19:54:11.686904 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.686883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.689704 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.689677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:54:11.689846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.689716    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 19:54:11.689846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.689684    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-smjhg\""
Sep 16 19:54:11.689846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.689762    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 19:54:11.689846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.689801    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 16 19:54:11.689846 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.689832    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 19:54:11.698422 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.698395    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"]
Sep 16 19:54:11.879828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.879781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-model-cache\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.879828 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.879828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-tmp-dir\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.880053 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.879876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4ws\" (UniqueName: \"kubernetes.io/projected/38fff1fe-9916-40f4-a9da-311cb681d726-kube-api-access-nv4ws\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.880053 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.879906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-kserve-provision-location\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.880053 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.879972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38fff1fe-9916-40f4-a9da-311cb681d726-cabundle-cert\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.880053 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.880015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-dshm\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.880190 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.880069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-home\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.880190 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.880090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38fff1fe-9916-40f4-a9da-311cb681d726-tls-certs\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.981528 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.981404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-kserve-provision-location\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.981528 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.981446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38fff1fe-9916-40f4-a9da-311cb681d726-cabundle-cert\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.981528 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.981474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-dshm\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.981837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.981562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-home\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.981837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.981587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38fff1fe-9916-40f4-a9da-311cb681d726-tls-certs\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.981837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.981636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-model-cache\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.981837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.981650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-tmp-dir\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.981837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.981669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4ws\" (UniqueName: \"kubernetes.io/projected/38fff1fe-9916-40f4-a9da-311cb681d726-kube-api-access-nv4ws\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.982091 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.981951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-kserve-provision-location\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.982091 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.982011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-home\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.982193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.982107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-tmp-dir\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.982193 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.982172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-model-cache\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.982305 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.982270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38fff1fe-9916-40f4-a9da-311cb681d726-cabundle-cert\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.983954 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.983930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-dshm\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.984136 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.984118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38fff1fe-9916-40f4-a9da-311cb681d726-tls-certs\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.994417 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.994392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4ws\" (UniqueName: \"kubernetes.io/projected/38fff1fe-9916-40f4-a9da-311cb681d726-kube-api-access-nv4ws\") pod \"tls-verification-test-kserve-7b8847c58b-ztjd4\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:11.998227 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:11.998205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:12.281670 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:12.281637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"]
Sep 16 19:54:12.283534 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:54:12.283504    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fff1fe_9916_40f4_a9da_311cb681d726.slice/crio-7420a1834868508e966527c90b5a04196a52477bb771ef5806c657e127172d96 WatchSource:0}: Error finding container 7420a1834868508e966527c90b5a04196a52477bb771ef5806c657e127172d96: Status 404 returned error can't find the container with id 7420a1834868508e966527c90b5a04196a52477bb771ef5806c657e127172d96
Sep 16 19:54:12.738017 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:12.737920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" event={"ID":"38fff1fe-9916-40f4-a9da-311cb681d726","Type":"ContainerStarted","Data":"d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612"}
Sep 16 19:54:12.738017 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:12.737976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" event={"ID":"38fff1fe-9916-40f4-a9da-311cb681d726","Type":"ContainerStarted","Data":"7420a1834868508e966527c90b5a04196a52477bb771ef5806c657e127172d96"}
Sep 16 19:54:14.748306 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:14.748267    2578 generic.go:356] "Generic (PLEG): container finished" podID="38fff1fe-9916-40f4-a9da-311cb681d726" containerID="d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612" exitCode=0
Sep 16 19:54:14.748732 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:14.748339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" event={"ID":"38fff1fe-9916-40f4-a9da-311cb681d726","Type":"ContainerDied","Data":"d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612"}
Sep 16 19:54:15.756922 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:15.756884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" event={"ID":"38fff1fe-9916-40f4-a9da-311cb681d726","Type":"ContainerStarted","Data":"af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54"}
Sep 16 19:54:15.815413 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:15.815356    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podStartSLOduration=2.600859088 podStartE2EDuration="4.815338121s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.214479033s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:15.814639866 +0000 UTC m=+4163.261734428" watchObservedRunningTime="2026-09-16 19:54:15.815338121 +0000 UTC m=+4163.262432650"
Sep 16 19:54:15.999337 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:15.999295    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:21.998440 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:21.998400    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:54:22.000207 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:22.000175    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:54:26.989058 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:26.989016    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"]
Sep 16 19:54:26.991859 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:26.991837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:54:26.996110 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:26.996075    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:54:26.996256 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:26.996075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:54:26.996256 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:26.996075    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:54:26.996256 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:26.996084    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-xxrsk\""
Sep 16 19:54:26.996256 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:26.996136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-9chcl\""
Sep 16 19:54:27.004650 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.004625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"]
Sep 16 19:54:27.118165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.118113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8q6\" (UniqueName: \"kubernetes.io/projected/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-kube-api-access-sf8q6\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469\" (UID: \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:54:27.118165 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.118164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469\" (UID: \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:54:27.219197 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.219144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8q6\" (UniqueName: \"kubernetes.io/projected/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-kube-api-access-sf8q6\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469\" (UID: \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:54:27.219197 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.219198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469\" (UID: \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:54:27.221703 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.221676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469\" (UID: \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:54:27.230432 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.230405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8q6\" (UniqueName: \"kubernetes.io/projected/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-kube-api-access-sf8q6\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469\" (UID: \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:54:27.303339 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.303286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:54:27.578598 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.578567    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"]
Sep 16 19:54:27.581318 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:54:27.581288    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode957f19c_aa5e_4114_89a7_cffc3d7bb58e.slice/crio-994cd25d306a2e72cf08b4078041cdd890d59f81d8cd1164763484b924168cbc WatchSource:0}: Error finding container 994cd25d306a2e72cf08b4078041cdd890d59f81d8cd1164763484b924168cbc: Status 404 returned error can't find the container with id 994cd25d306a2e72cf08b4078041cdd890d59f81d8cd1164763484b924168cbc
Sep 16 19:54:27.820961 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.820923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" event={"ID":"e957f19c-aa5e-4114-89a7-cffc3d7bb58e","Type":"ContainerStarted","Data":"d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a"}
Sep 16 19:54:27.821145 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.820985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" event={"ID":"e957f19c-aa5e-4114-89a7-cffc3d7bb58e","Type":"ContainerStarted","Data":"994cd25d306a2e72cf08b4078041cdd890d59f81d8cd1164763484b924168cbc"}
Sep 16 19:54:27.821145 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.821034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:54:27.842445 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:27.842334    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" podStartSLOduration=1.8423127080000001 podStartE2EDuration="1.842312708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:27.840211337 +0000 UTC m=+4175.287305889" watchObservedRunningTime="2026-09-16 19:54:27.842312708 +0000 UTC m=+4175.289407262"
Sep 16 19:54:31.999400 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:31.999349    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:54:41.999323 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:41.999273    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:54:51.998809 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:51.998763    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:54:53.360205 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:53.360175    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:54:53.364104 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:53.364079    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:54:53.385944 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:53.385918    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:54:53.389688 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:53.389666    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:54:58.828989 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:54:58.828951    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:55:01.999660 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:55:01.999608    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:55:11.999540 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:55:11.999467    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:55:21.998935 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:55:21.998830    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:55:31.998928 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:55:31.998871    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:55:41.998884 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:55:41.998835    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:55:51.999621 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:55:51.999561    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:56:01.999701 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:01.999645    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 19:56:12.009359 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:12.009320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:56:12.017900 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:12.017873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:56:17.007617 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:17.007569    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"]
Sep 16 19:56:32.030601 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.030558    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main" containerID="cri-o://af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54" gracePeriod=45
Sep 16 19:56:32.778909 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.778880    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:56:32.849031 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.848941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-tmp-dir\") pod \"38fff1fe-9916-40f4-a9da-311cb681d726\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") "
Sep 16 19:56:32.849031 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.848994    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-kserve-provision-location\") pod \"38fff1fe-9916-40f4-a9da-311cb681d726\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") "
Sep 16 19:56:32.849031 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38fff1fe-9916-40f4-a9da-311cb681d726-cabundle-cert\") pod \"38fff1fe-9916-40f4-a9da-311cb681d726\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") "
Sep 16 19:56:32.849333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849074    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-home\") pod \"38fff1fe-9916-40f4-a9da-311cb681d726\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") "
Sep 16 19:56:32.849333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-model-cache\") pod \"38fff1fe-9916-40f4-a9da-311cb681d726\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") "
Sep 16 19:56:32.849333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-dshm\") pod \"38fff1fe-9916-40f4-a9da-311cb681d726\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") "
Sep 16 19:56:32.849333 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv4ws\" (UniqueName: \"kubernetes.io/projected/38fff1fe-9916-40f4-a9da-311cb681d726-kube-api-access-nv4ws\") pod \"38fff1fe-9916-40f4-a9da-311cb681d726\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") "
Sep 16 19:56:32.849599 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849341    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-model-cache" (OuterVolumeSpecName: "model-cache") pod "38fff1fe-9916-40f4-a9da-311cb681d726" (UID: "38fff1fe-9916-40f4-a9da-311cb681d726"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:32.849599 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849378    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38fff1fe-9916-40f4-a9da-311cb681d726-tls-certs\") pod \"38fff1fe-9916-40f4-a9da-311cb681d726\" (UID: \"38fff1fe-9916-40f4-a9da-311cb681d726\") "
Sep 16 19:56:32.849599 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fff1fe-9916-40f4-a9da-311cb681d726-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38fff1fe-9916-40f4-a9da-311cb681d726" (UID: "38fff1fe-9916-40f4-a9da-311cb681d726"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:56:32.849746 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849688    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:56:32.849746 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849722    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38fff1fe-9916-40f4-a9da-311cb681d726-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:56:32.849746 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.849721    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-home" (OuterVolumeSpecName: "home") pod "38fff1fe-9916-40f4-a9da-311cb681d726" (UID: "38fff1fe-9916-40f4-a9da-311cb681d726"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:32.851581 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.851543    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-dshm" (OuterVolumeSpecName: "dshm") pod "38fff1fe-9916-40f4-a9da-311cb681d726" (UID: "38fff1fe-9916-40f4-a9da-311cb681d726"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:32.851707 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.851607    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38fff1fe-9916-40f4-a9da-311cb681d726-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38fff1fe-9916-40f4-a9da-311cb681d726" (UID: "38fff1fe-9916-40f4-a9da-311cb681d726"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:32.851772 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.851723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fff1fe-9916-40f4-a9da-311cb681d726-kube-api-access-nv4ws" (OuterVolumeSpecName: "kube-api-access-nv4ws") pod "38fff1fe-9916-40f4-a9da-311cb681d726" (UID: "38fff1fe-9916-40f4-a9da-311cb681d726"). InnerVolumeSpecName "kube-api-access-nv4ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:32.866351 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.866290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38fff1fe-9916-40f4-a9da-311cb681d726" (UID: "38fff1fe-9916-40f4-a9da-311cb681d726"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:32.874331 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.874303    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38fff1fe-9916-40f4-a9da-311cb681d726" (UID: "38fff1fe-9916-40f4-a9da-311cb681d726"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:32.950137 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.950099    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:56:32.950137 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.950130    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:56:32.950137 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.950139    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:56:32.950137 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.950147    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38fff1fe-9916-40f4-a9da-311cb681d726-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:56:32.950409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.950156    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv4ws\" (UniqueName: \"kubernetes.io/projected/38fff1fe-9916-40f4-a9da-311cb681d726-kube-api-access-nv4ws\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:56:32.950409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:32.950165    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38fff1fe-9916-40f4-a9da-311cb681d726-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:56:33.462692 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.462654    2578 generic.go:356] "Generic (PLEG): container finished" podID="38fff1fe-9916-40f4-a9da-311cb681d726" containerID="af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54" exitCode=0
Sep 16 19:56:33.463111 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.462705    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"
Sep 16 19:56:33.463111 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.462727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" event={"ID":"38fff1fe-9916-40f4-a9da-311cb681d726","Type":"ContainerDied","Data":"af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54"}
Sep 16 19:56:33.463111 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.462779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4" event={"ID":"38fff1fe-9916-40f4-a9da-311cb681d726","Type":"ContainerDied","Data":"7420a1834868508e966527c90b5a04196a52477bb771ef5806c657e127172d96"}
Sep 16 19:56:33.463111 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.462801    2578 scope.go:117] "RemoveContainer" containerID="af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54"
Sep 16 19:56:33.471593 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.471567    2578 scope.go:117] "RemoveContainer" containerID="d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612"
Sep 16 19:56:33.479950 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.479924    2578 scope.go:117] "RemoveContainer" containerID="af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54"
Sep 16 19:56:33.480245 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:56:33.480225    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54\": container with ID starting with af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54 not found: ID does not exist" containerID="af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54"
Sep 16 19:56:33.480304 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.480256    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54"} err="failed to get container status \"af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54\": rpc error: code = NotFound desc = could not find container \"af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54\": container with ID starting with af978b79b84737490e8414bb79d9e3c96b08ee56c28a190d7f09a3f2697ccc54 not found: ID does not exist"
Sep 16 19:56:33.480304 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.480275    2578 scope.go:117] "RemoveContainer" containerID="d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612"
Sep 16 19:56:33.480650 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:56:33.480624    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612\": container with ID starting with d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612 not found: ID does not exist" containerID="d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612"
Sep 16 19:56:33.480734 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.480658    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612"} err="failed to get container status \"d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612\": rpc error: code = NotFound desc = could not find container \"d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612\": container with ID starting with d0dad4e294f2b11655823eb55126904f26fa51dfd06308988419df5740f2a612 not found: ID does not exist"
Sep 16 19:56:33.483507 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.483470    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"]
Sep 16 19:56:33.487101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:33.487076    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7b8847c58b-ztjd4"]
Sep 16 19:56:35.063494 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:35.063439    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" path="/var/lib/kubelet/pods/38fff1fe-9916-40f4-a9da-311cb681d726/volumes"
Sep 16 19:56:36.315101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:36.315062    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"]
Sep 16 19:56:38.828961 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:38.828919    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:48.828785 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:48.828741    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:51.334301 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:51.334241    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerName="main" containerID="cri-o://d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a" gracePeriod=45
Sep 16 19:56:57.132085 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:57.132046    2578 scope.go:117] "RemoveContainer" containerID="abdf062a73db2c07c75f221c70977541f2cad1868337a2966038edd36137dd3b"
Sep 16 19:56:58.828358 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:58.828319    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:58.828772 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:56:58.828429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:57:08.828847 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:08.828804    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:18.828664 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:18.828618    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:21.582964 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.582935    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:57:21.682532 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.682396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf8q6\" (UniqueName: \"kubernetes.io/projected/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-kube-api-access-sf8q6\") pod \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\" (UID: \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\") "
Sep 16 19:57:21.682532 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.682527    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-tls-certs\") pod \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\" (UID: \"e957f19c-aa5e-4114-89a7-cffc3d7bb58e\") "
Sep 16 19:57:21.684749 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.684727    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e957f19c-aa5e-4114-89a7-cffc3d7bb58e" (UID: "e957f19c-aa5e-4114-89a7-cffc3d7bb58e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:21.684869 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.684740    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-kube-api-access-sf8q6" (OuterVolumeSpecName: "kube-api-access-sf8q6") pod "e957f19c-aa5e-4114-89a7-cffc3d7bb58e" (UID: "e957f19c-aa5e-4114-89a7-cffc3d7bb58e"). InnerVolumeSpecName "kube-api-access-sf8q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:21.693285 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.693256    2578 generic.go:356] "Generic (PLEG): container finished" podID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerID="d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a" exitCode=0
Sep 16 19:57:21.693409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.693298    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"
Sep 16 19:57:21.693409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.693306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" event={"ID":"e957f19c-aa5e-4114-89a7-cffc3d7bb58e","Type":"ContainerDied","Data":"d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a"}
Sep 16 19:57:21.693409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.693348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469" event={"ID":"e957f19c-aa5e-4114-89a7-cffc3d7bb58e","Type":"ContainerDied","Data":"994cd25d306a2e72cf08b4078041cdd890d59f81d8cd1164763484b924168cbc"}
Sep 16 19:57:21.693409 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.693369    2578 scope.go:117] "RemoveContainer" containerID="d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a"
Sep 16 19:57:21.703684 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.703655    2578 scope.go:117] "RemoveContainer" containerID="d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a"
Sep 16 19:57:21.704052 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:57:21.704025    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a\": container with ID starting with d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a not found: ID does not exist" containerID="d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a"
Sep 16 19:57:21.704116 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.704062    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a"} err="failed to get container status \"d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a\": rpc error: code = NotFound desc = could not find container \"d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a\": container with ID starting with d7209fe498ae23b3985eb9efea557351f0d0d816711011b2f45c53815fd4008a not found: ID does not exist"
Sep 16 19:57:21.720489 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.720444    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"]
Sep 16 19:57:21.725414 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.725386    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48b8469"]
Sep 16 19:57:21.735464 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:57:21.735435    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode957f19c_aa5e_4114_89a7_cffc3d7bb58e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode957f19c_aa5e_4114_89a7_cffc3d7bb58e.slice/crio-994cd25d306a2e72cf08b4078041cdd890d59f81d8cd1164763484b924168cbc\": RecentStats: unable to find data in memory cache]"
Sep 16 19:57:21.783262 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.783213    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:57:21.783262 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:21.783255    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf8q6\" (UniqueName: \"kubernetes.io/projected/e957f19c-aa5e-4114-89a7-cffc3d7bb58e-kube-api-access-sf8q6\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:57:23.064657 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:23.064623    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" path="/var/lib/kubelet/pods/e957f19c-aa5e-4114-89a7-cffc3d7bb58e/volumes"
Sep 16 19:57:25.597665 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.597623    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"]
Sep 16 19:57:25.598053 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.598041    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerName="main"
Sep 16 19:57:25.598101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.598055    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerName="main"
Sep 16 19:57:25.598101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.598074    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="storage-initializer"
Sep 16 19:57:25.598101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.598080    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="storage-initializer"
Sep 16 19:57:25.598101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.598094    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main"
Sep 16 19:57:25.598101 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.598099    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main"
Sep 16 19:57:25.598257 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.598155    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="38fff1fe-9916-40f4-a9da-311cb681d726" containerName="main"
Sep 16 19:57:25.598257 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.598163    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e957f19c-aa5e-4114-89a7-cffc3d7bb58e" containerName="main"
Sep 16 19:57:25.602602 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.602580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.606555 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.606526    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 19:57:25.606555 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.606553    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 19:57:25.606755 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.606579    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 19:57:25.606860 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.606845    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-mbzfw\""
Sep 16 19:57:25.606991 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.606975    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 19:57:25.607392 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.607379    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:57:25.623837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.623801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"]
Sep 16 19:57:25.712676 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.712639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-dshm\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.712844 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.712683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tls-certs\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.712844 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.712732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-cabundle-cert\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.712844 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.712813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kube-api-access-hnwzk\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.712979 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.712851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-home\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.712979 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.712871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-model-cache\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.712979 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.712962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kserve-provision-location\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.713082 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.713010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tmp-dir\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814080 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-home\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814278 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-model-cache\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814278 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kserve-provision-location\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814278 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tmp-dir\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814278 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-dshm\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814278 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tls-certs\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814278 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-cabundle-cert\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814278 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kube-api-access-hnwzk\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814655 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-home\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814732 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-model-cache\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814830 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tmp-dir\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.814953 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.814837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kserve-provision-location\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.815129 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.815108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-cabundle-cert\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.816754 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.816736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-dshm\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.816908 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.816891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tls-certs\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.824092 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.824067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kube-api-access-hnwzk\") pod \"prestop-hook-test-kserve-685bf59966-2cblz\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:25.913807 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:25.913722    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:26.196762 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:26.196732    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"]
Sep 16 19:57:26.198375 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:57:26.198348    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b3c8b1_44b4_4796_82aa_6b85d45a4435.slice/crio-e2b77c2d3a9b85604eee692c7d471c03ad50c96d9cee708757b510c6b4a4734f WatchSource:0}: Error finding container e2b77c2d3a9b85604eee692c7d471c03ad50c96d9cee708757b510c6b4a4734f: Status 404 returned error can't find the container with id e2b77c2d3a9b85604eee692c7d471c03ad50c96d9cee708757b510c6b4a4734f
Sep 16 19:57:26.721380 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:26.721340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" event={"ID":"d4b3c8b1-44b4-4796-82aa-6b85d45a4435","Type":"ContainerStarted","Data":"76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0"}
Sep 16 19:57:26.721769 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:26.721409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" event={"ID":"d4b3c8b1-44b4-4796-82aa-6b85d45a4435","Type":"ContainerStarted","Data":"e2b77c2d3a9b85604eee692c7d471c03ad50c96d9cee708757b510c6b4a4734f"}
Sep 16 19:57:28.735493 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:28.735433    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerID="76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0" exitCode=0
Sep 16 19:57:28.735853 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:28.735503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" event={"ID":"d4b3c8b1-44b4-4796-82aa-6b85d45a4435","Type":"ContainerDied","Data":"76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0"}
Sep 16 19:57:29.743224 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:29.743189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" event={"ID":"d4b3c8b1-44b4-4796-82aa-6b85d45a4435","Type":"ContainerStarted","Data":"fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03"}
Sep 16 19:57:29.777433 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:29.777373    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podStartSLOduration=2.5305487490000003 podStartE2EDuration="4.777352798s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.246804049s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:29.776666921 +0000 UTC m=+4357.223761453" watchObservedRunningTime="2026-09-16 19:57:29.777352798 +0000 UTC m=+4357.224447329"
Sep 16 19:57:29.914291 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:29.914251    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:35.914492 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:35.914445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:57:35.916241 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:35.916207    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:57:45.914594 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:45.914538    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:57:55.914653 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:57:55.914608    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:58:05.914589 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:05.914534    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:58:13.335105 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.335064    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"]
Sep 16 19:58:13.339837 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.339814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:13.344345 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.344313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-ttkd6\""
Sep 16 19:58:13.344493 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.344340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:58:13.344493 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.344366    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:58:13.344493 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.344318    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:58:13.344493 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.344313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-pkq29\""
Sep 16 19:58:13.350902 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.350877    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"]
Sep 16 19:58:13.440932 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.440892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114570c7-b869-4ce7-b6fd-38140e465cbc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl\" (UID: \"114570c7-b869-4ce7-b6fd-38140e465cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:13.441109 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.440958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6ds\" (UniqueName: \"kubernetes.io/projected/114570c7-b869-4ce7-b6fd-38140e465cbc-kube-api-access-xx6ds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl\" (UID: \"114570c7-b869-4ce7-b6fd-38140e465cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:13.542138 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.542100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6ds\" (UniqueName: \"kubernetes.io/projected/114570c7-b869-4ce7-b6fd-38140e465cbc-kube-api-access-xx6ds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl\" (UID: \"114570c7-b869-4ce7-b6fd-38140e465cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:13.542326 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.542209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114570c7-b869-4ce7-b6fd-38140e465cbc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl\" (UID: \"114570c7-b869-4ce7-b6fd-38140e465cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:13.544803 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.544777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114570c7-b869-4ce7-b6fd-38140e465cbc-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl\" (UID: \"114570c7-b869-4ce7-b6fd-38140e465cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:13.551285 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.551258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6ds\" (UniqueName: \"kubernetes.io/projected/114570c7-b869-4ce7-b6fd-38140e465cbc-kube-api-access-xx6ds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl\" (UID: \"114570c7-b869-4ce7-b6fd-38140e465cbc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:13.651271 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.651177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:13.934258 ip-10-0-143-161 kubenswrapper[2578]: W0916 19:58:13.930783    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114570c7_b869_4ce7_b6fd_38140e465cbc.slice/crio-7a8542a4b9d06bcb769079a7cd6e0ebca08e8987a40978055f7cbd54c2a365f5 WatchSource:0}: Error finding container 7a8542a4b9d06bcb769079a7cd6e0ebca08e8987a40978055f7cbd54c2a365f5: Status 404 returned error can't find the container with id 7a8542a4b9d06bcb769079a7cd6e0ebca08e8987a40978055f7cbd54c2a365f5
Sep 16 19:58:13.934258 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.933389    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"]
Sep 16 19:58:13.978637 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:13.978527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" event={"ID":"114570c7-b869-4ce7-b6fd-38140e465cbc","Type":"ContainerStarted","Data":"7a8542a4b9d06bcb769079a7cd6e0ebca08e8987a40978055f7cbd54c2a365f5"}
Sep 16 19:58:14.985749 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:14.985707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" event={"ID":"114570c7-b869-4ce7-b6fd-38140e465cbc","Type":"ContainerStarted","Data":"33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b"}
Sep 16 19:58:14.986145 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:14.985844    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:15.006447 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:15.006395    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" podStartSLOduration=2.006379735 podStartE2EDuration="2.006379735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:15.004086098 +0000 UTC m=+4402.451180647" watchObservedRunningTime="2026-09-16 19:58:15.006379735 +0000 UTC m=+4402.453474264"
Sep 16 19:58:15.915047 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:15.914998    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:58:25.914268 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:25.914217    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:58:35.915195 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:35.915138    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:58:45.915040 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:45.914997    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:58:45.993247 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:45.993216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 19:58:55.915252 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:58:55.915202    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:59:05.914971 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:05.914915    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:59:15.914398 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:15.914349    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 19:59:25.924015 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:25.923974    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:59:25.931821 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:25.931786    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:59:28.081424 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:28.081386    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"]
Sep 16 19:59:43.104078 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.104030    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main" containerID="cri-o://fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03" gracePeriod=45
Sep 16 19:59:43.850777 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.850749    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:59:43.951653 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.951547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-model-cache\") pod \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") "
Sep 16 19:59:43.951653 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.951598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-dshm\") pod \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") "
Sep 16 19:59:43.951653 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.951619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tls-certs\") pod \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") "
Sep 16 19:59:43.951653 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.951664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-home\") pod \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") "
Sep 16 19:59:43.951996 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.951696    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tmp-dir\") pod \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") "
Sep 16 19:59:43.951996 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.951718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kserve-provision-location\") pod \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") "
Sep 16 19:59:43.951996 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.951755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kube-api-access-hnwzk\") pod \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") "
Sep 16 19:59:43.951996 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.951787    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-cabundle-cert\") pod \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\" (UID: \"d4b3c8b1-44b4-4796-82aa-6b85d45a4435\") "
Sep 16 19:59:43.951996 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.951864    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4b3c8b1-44b4-4796-82aa-6b85d45a4435" (UID: "d4b3c8b1-44b4-4796-82aa-6b85d45a4435"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:43.952267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.952092    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:59:43.952267 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.952205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4b3c8b1-44b4-4796-82aa-6b85d45a4435" (UID: "d4b3c8b1-44b4-4796-82aa-6b85d45a4435"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:59:43.952432 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.952400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-home" (OuterVolumeSpecName: "home") pod "d4b3c8b1-44b4-4796-82aa-6b85d45a4435" (UID: "d4b3c8b1-44b4-4796-82aa-6b85d45a4435"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:43.954616 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.954586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-dshm" (OuterVolumeSpecName: "dshm") pod "d4b3c8b1-44b4-4796-82aa-6b85d45a4435" (UID: "d4b3c8b1-44b4-4796-82aa-6b85d45a4435"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:43.954728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.954591    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4b3c8b1-44b4-4796-82aa-6b85d45a4435" (UID: "d4b3c8b1-44b4-4796-82aa-6b85d45a4435"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:59:43.954728 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.954701    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kube-api-access-hnwzk" (OuterVolumeSpecName: "kube-api-access-hnwzk") pod "d4b3c8b1-44b4-4796-82aa-6b85d45a4435" (UID: "d4b3c8b1-44b4-4796-82aa-6b85d45a4435"). InnerVolumeSpecName "kube-api-access-hnwzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:59:43.964541 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.964492    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4b3c8b1-44b4-4796-82aa-6b85d45a4435" (UID: "d4b3c8b1-44b4-4796-82aa-6b85d45a4435"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:43.976266 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:43.976234    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4b3c8b1-44b4-4796-82aa-6b85d45a4435" (UID: "d4b3c8b1-44b4-4796-82aa-6b85d45a4435"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:59:44.053294 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.053264    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:59:44.053294 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.053293    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:59:44.053516 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.053304    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:59:44.053516 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.053319    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-kube-api-access-hnwzk\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:59:44.053516 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.053331    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:59:44.053516 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.053339    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:59:44.053516 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.053346    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3c8b1-44b4-4796-82aa-6b85d45a4435-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 19:59:44.439115 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.439076    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerID="fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03" exitCode=0
Sep 16 19:59:44.439524 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.439129    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"
Sep 16 19:59:44.439524 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.439153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" event={"ID":"d4b3c8b1-44b4-4796-82aa-6b85d45a4435","Type":"ContainerDied","Data":"fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03"}
Sep 16 19:59:44.439524 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.439212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz" event={"ID":"d4b3c8b1-44b4-4796-82aa-6b85d45a4435","Type":"ContainerDied","Data":"e2b77c2d3a9b85604eee692c7d471c03ad50c96d9cee708757b510c6b4a4734f"}
Sep 16 19:59:44.439524 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.439245    2578 scope.go:117] "RemoveContainer" containerID="fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03"
Sep 16 19:59:44.448666 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.448645    2578 scope.go:117] "RemoveContainer" containerID="76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0"
Sep 16 19:59:44.457128 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.457107    2578 scope.go:117] "RemoveContainer" containerID="fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03"
Sep 16 19:59:44.457385 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:59:44.457363    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03\": container with ID starting with fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03 not found: ID does not exist" containerID="fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03"
Sep 16 19:59:44.457436 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.457396    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03"} err="failed to get container status \"fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03\": rpc error: code = NotFound desc = could not find container \"fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03\": container with ID starting with fb923e3ac971e091de06507b57f810263bdc0ff104719a17c4424cf77f3b3f03 not found: ID does not exist"
Sep 16 19:59:44.457436 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.457420    2578 scope.go:117] "RemoveContainer" containerID="76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0"
Sep 16 19:59:44.457734 ip-10-0-143-161 kubenswrapper[2578]: E0916 19:59:44.457715    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0\": container with ID starting with 76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0 not found: ID does not exist" containerID="76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0"
Sep 16 19:59:44.457793 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.457740    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0"} err="failed to get container status \"76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0\": rpc error: code = NotFound desc = could not find container \"76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0\": container with ID starting with 76d42fce8b92fdae120138a18bf6f84cb41e0b0c301c9dde118627bfaa2d7ef0 not found: ID does not exist"
Sep 16 19:59:44.473729 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.473699    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"]
Sep 16 19:59:44.493112 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:44.488180    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-2cblz"]
Sep 16 19:59:45.064124 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:45.064088    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" path="/var/lib/kubelet/pods/d4b3c8b1-44b4-4796-82aa-6b85d45a4435/volumes"
Sep 16 19:59:53.390734 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:53.390706    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:59:53.394709 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:53.394685    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 19:59:53.419454 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:53.419426    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 19:59:53.423043 ip-10-0-143-161 kubenswrapper[2578]: I0916 19:59:53.423023    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 20:00:22.885037 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:00:22.884999    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"]
Sep 16 20:00:25.992547 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:00:25.992507    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:35.992745 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:00:35.992702    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:37.904673 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:00:37.904610    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerName="main" containerID="cri-o://33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b" gracePeriod=45
Sep 16 20:00:45.992472 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:00:45.992429    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:45.992868 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:00:45.992560    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 20:00:55.999141 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:00:55.997644    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:05.993417 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:05.993377    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:08.155176 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.155149    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 20:01:08.266530 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.266409    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114570c7-b869-4ce7-b6fd-38140e465cbc-tls-certs\") pod \"114570c7-b869-4ce7-b6fd-38140e465cbc\" (UID: \"114570c7-b869-4ce7-b6fd-38140e465cbc\") "
Sep 16 20:01:08.266530 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.266495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx6ds\" (UniqueName: \"kubernetes.io/projected/114570c7-b869-4ce7-b6fd-38140e465cbc-kube-api-access-xx6ds\") pod \"114570c7-b869-4ce7-b6fd-38140e465cbc\" (UID: \"114570c7-b869-4ce7-b6fd-38140e465cbc\") "
Sep 16 20:01:08.268732 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.268704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114570c7-b869-4ce7-b6fd-38140e465cbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "114570c7-b869-4ce7-b6fd-38140e465cbc" (UID: "114570c7-b869-4ce7-b6fd-38140e465cbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:08.268833 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.268775    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114570c7-b869-4ce7-b6fd-38140e465cbc-kube-api-access-xx6ds" (OuterVolumeSpecName: "kube-api-access-xx6ds") pod "114570c7-b869-4ce7-b6fd-38140e465cbc" (UID: "114570c7-b869-4ce7-b6fd-38140e465cbc"). InnerVolumeSpecName "kube-api-access-xx6ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:08.367470 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.367428    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/114570c7-b869-4ce7-b6fd-38140e465cbc-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:01:08.367470 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.367463    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx6ds\" (UniqueName: \"kubernetes.io/projected/114570c7-b869-4ce7-b6fd-38140e465cbc-kube-api-access-xx6ds\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:01:08.837269 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.837228    2578 generic.go:356] "Generic (PLEG): container finished" podID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerID="33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b" exitCode=0
Sep 16 20:01:08.837269 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.837274    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"
Sep 16 20:01:08.837573 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.837318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" event={"ID":"114570c7-b869-4ce7-b6fd-38140e465cbc","Type":"ContainerDied","Data":"33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b"}
Sep 16 20:01:08.837573 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.837384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl" event={"ID":"114570c7-b869-4ce7-b6fd-38140e465cbc","Type":"ContainerDied","Data":"7a8542a4b9d06bcb769079a7cd6e0ebca08e8987a40978055f7cbd54c2a365f5"}
Sep 16 20:01:08.837573 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.837415    2578 scope.go:117] "RemoveContainer" containerID="33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b"
Sep 16 20:01:08.846407 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.846388    2578 scope.go:117] "RemoveContainer" containerID="33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b"
Sep 16 20:01:08.846706 ip-10-0-143-161 kubenswrapper[2578]: E0916 20:01:08.846684    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b\": container with ID starting with 33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b not found: ID does not exist" containerID="33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b"
Sep 16 20:01:08.846752 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.846716    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b"} err="failed to get container status \"33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b\": rpc error: code = NotFound desc = could not find container \"33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b\": container with ID starting with 33340e93e073f233946571179393aa2ba79815fbf5dc2b7846292f4b8c7c033b not found: ID does not exist"
Sep 16 20:01:08.861716 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.861681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"]
Sep 16 20:01:08.869831 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:08.869800    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454h7rl"]
Sep 16 20:01:09.063703 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:09.063660    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" path="/var/lib/kubelet/pods/114570c7-b869-4ce7-b6fd-38140e465cbc/volumes"
Sep 16 20:01:34.377221 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.377181    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"]
Sep 16 20:01:34.378239 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.377577    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="storage-initializer"
Sep 16 20:01:34.378239 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.377590    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="storage-initializer"
Sep 16 20:01:34.378239 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.377600    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main"
Sep 16 20:01:34.378239 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.377607    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main"
Sep 16 20:01:34.378239 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.377621    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerName="main"
Sep 16 20:01:34.378239 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.377627    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerName="main"
Sep 16 20:01:34.378239 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.377685    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="114570c7-b869-4ce7-b6fd-38140e465cbc" containerName="main"
Sep 16 20:01:34.378239 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.377693    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b3c8b1-44b4-4796-82aa-6b85d45a4435" containerName="main"
Sep 16 20:01:34.380804 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.380785    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.383708 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.383686    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-tlwtc\""
Sep 16 20:01:34.383818 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.383693    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 20:01:34.384360 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.384344    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 20:01:34.384409 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.384390    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:01:34.384645 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.384627    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 20:01:34.384931 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.384919    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 20:01:34.402387 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.402351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m8hx\" (UniqueName: \"kubernetes.io/projected/f8ce49b5-db92-4930-9f44-4f341b190bde-kube-api-access-9m8hx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.402590 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.402406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce49b5-db92-4930-9f44-4f341b190bde-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.402590 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.402435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.402590 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.402465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ce49b5-db92-4930-9f44-4f341b190bde-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.402590 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.402507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.402590 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.402524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.402590 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.402542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.402917 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.402597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.402917 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.402840    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"]
Sep 16 20:01:34.503668 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.503626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ce49b5-db92-4930-9f44-4f341b190bde-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.503668 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.503668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.503987 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.503689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.503987 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.503716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.503987 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.503759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.503987 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.503825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m8hx\" (UniqueName: \"kubernetes.io/projected/f8ce49b5-db92-4930-9f44-4f341b190bde-kube-api-access-9m8hx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.503987 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.503873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce49b5-db92-4930-9f44-4f341b190bde-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.503987 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.503900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.504325 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.504145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.504325 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.504197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.504325 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.504256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.504575 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.504552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.504703 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.504687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce49b5-db92-4930-9f44-4f341b190bde-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.506163 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.506134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.506393 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.506377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ce49b5-db92-4930-9f44-4f341b190bde-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.513412 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.513384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m8hx\" (UniqueName: \"kubernetes.io/projected/f8ce49b5-db92-4930-9f44-4f341b190bde-kube-api-access-9m8hx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.690791 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.690689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:34.969740 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:34.969710    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"]
Sep 16 20:01:34.971844 ip-10-0-143-161 kubenswrapper[2578]: W0916 20:01:34.971819    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ce49b5_db92_4930_9f44_4f341b190bde.slice/crio-2e441227091b867b34a9896931aff0d4cfd600c07a9d8d15a813367bca26261c WatchSource:0}: Error finding container 2e441227091b867b34a9896931aff0d4cfd600c07a9d8d15a813367bca26261c: Status 404 returned error can't find the container with id 2e441227091b867b34a9896931aff0d4cfd600c07a9d8d15a813367bca26261c
Sep 16 20:01:35.967853 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:35.967800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" event={"ID":"f8ce49b5-db92-4930-9f44-4f341b190bde","Type":"ContainerStarted","Data":"a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f"}
Sep 16 20:01:35.968366 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:35.967877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" event={"ID":"f8ce49b5-db92-4930-9f44-4f341b190bde","Type":"ContainerStarted","Data":"2e441227091b867b34a9896931aff0d4cfd600c07a9d8d15a813367bca26261c"}
Sep 16 20:01:37.988122 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:37.988076    2578 generic.go:356] "Generic (PLEG): container finished" podID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerID="a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f" exitCode=0
Sep 16 20:01:37.988524 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:37.988140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" event={"ID":"f8ce49b5-db92-4930-9f44-4f341b190bde","Type":"ContainerDied","Data":"a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f"}
Sep 16 20:01:38.994866 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:38.994828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" event={"ID":"f8ce49b5-db92-4930-9f44-4f341b190bde","Type":"ContainerStarted","Data":"10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92"}
Sep 16 20:01:39.025686 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:39.025620    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podStartSLOduration=2.804201127 podStartE2EDuration="5.025599715s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.221398588s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:39.024198564 +0000 UTC m=+4606.471293113" watchObservedRunningTime="2026-09-16 20:01:39.025599715 +0000 UTC m=+4606.472694246"
Sep 16 20:01:39.691102 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:39.691003    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:44.691134 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:44.691094    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:01:44.692743 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:44.692702    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:01:54.692039 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:01:54.691986    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:02:04.691542 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:02:04.691460    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:02:14.691246 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:02:14.691195    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:02:24.691783 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:02:24.691732    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:02:34.691582 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:02:34.691530    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:02:44.691727 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:02:44.691681    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:02:54.691523 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:02:54.691449    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:03:04.691367 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:04.691318    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:03:14.691826 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:14.691772    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:03:24.692148 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:24.692089    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:03:34.701134 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:34.701094    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:03:34.709001 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:34.708969    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:03:39.306279 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:39.306234    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"]
Sep 16 20:03:54.329207 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:54.329146    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" containerID="cri-o://10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92" gracePeriod=45
Sep 16 20:03:54.701959 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:54.701854    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 20:03:55.004864 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.004837    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:03:55.140138 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-model-cache\") pod \"f8ce49b5-db92-4930-9f44-4f341b190bde\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") "
Sep 16 20:03:55.140339 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ce49b5-db92-4930-9f44-4f341b190bde-tls-certs\") pod \"f8ce49b5-db92-4930-9f44-4f341b190bde\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") "
Sep 16 20:03:55.140339 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-dshm\") pod \"f8ce49b5-db92-4930-9f44-4f341b190bde\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") "
Sep 16 20:03:55.140339 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-home\") pod \"f8ce49b5-db92-4930-9f44-4f341b190bde\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") "
Sep 16 20:03:55.140339 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m8hx\" (UniqueName: \"kubernetes.io/projected/f8ce49b5-db92-4930-9f44-4f341b190bde-kube-api-access-9m8hx\") pod \"f8ce49b5-db92-4930-9f44-4f341b190bde\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") "
Sep 16 20:03:55.140339 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140284    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-tmp-dir\") pod \"f8ce49b5-db92-4930-9f44-4f341b190bde\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") "
Sep 16 20:03:55.140339 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-kserve-provision-location\") pod \"f8ce49b5-db92-4930-9f44-4f341b190bde\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") "
Sep 16 20:03:55.140723 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140367    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce49b5-db92-4930-9f44-4f341b190bde-cabundle-cert\") pod \"f8ce49b5-db92-4930-9f44-4f341b190bde\" (UID: \"f8ce49b5-db92-4930-9f44-4f341b190bde\") "
Sep 16 20:03:55.140723 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140418    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8ce49b5-db92-4930-9f44-4f341b190bde" (UID: "f8ce49b5-db92-4930-9f44-4f341b190bde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:55.140723 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140660    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-model-cache\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:03:55.141018 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.140977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ce49b5-db92-4930-9f44-4f341b190bde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f8ce49b5-db92-4930-9f44-4f341b190bde" (UID: "f8ce49b5-db92-4930-9f44-4f341b190bde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:03:55.141152 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.141027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-home" (OuterVolumeSpecName: "home") pod "f8ce49b5-db92-4930-9f44-4f341b190bde" (UID: "f8ce49b5-db92-4930-9f44-4f341b190bde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:55.143023 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.142987    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ce49b5-db92-4930-9f44-4f341b190bde-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8ce49b5-db92-4930-9f44-4f341b190bde" (UID: "f8ce49b5-db92-4930-9f44-4f341b190bde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:55.143140 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.143047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ce49b5-db92-4930-9f44-4f341b190bde-kube-api-access-9m8hx" (OuterVolumeSpecName: "kube-api-access-9m8hx") pod "f8ce49b5-db92-4930-9f44-4f341b190bde" (UID: "f8ce49b5-db92-4930-9f44-4f341b190bde"). InnerVolumeSpecName "kube-api-access-9m8hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:55.143140 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.143057    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-dshm" (OuterVolumeSpecName: "dshm") pod "f8ce49b5-db92-4930-9f44-4f341b190bde" (UID: "f8ce49b5-db92-4930-9f44-4f341b190bde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:55.152935 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.152902    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8ce49b5-db92-4930-9f44-4f341b190bde" (UID: "f8ce49b5-db92-4930-9f44-4f341b190bde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:55.165689 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.165652    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8ce49b5-db92-4930-9f44-4f341b190bde" (UID: "f8ce49b5-db92-4930-9f44-4f341b190bde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:55.241446 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.241353    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ce49b5-db92-4930-9f44-4f341b190bde-tls-certs\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:03:55.241446 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.241384    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-dshm\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:03:55.241446 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.241392    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-home\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:03:55.241446 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.241401    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m8hx\" (UniqueName: \"kubernetes.io/projected/f8ce49b5-db92-4930-9f44-4f341b190bde-kube-api-access-9m8hx\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:03:55.241446 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.241411    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-tmp-dir\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:03:55.241446 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.241421    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8ce49b5-db92-4930-9f44-4f341b190bde-kserve-provision-location\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:03:55.241446 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.241430    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8ce49b5-db92-4930-9f44-4f341b190bde-cabundle-cert\") on node \"ip-10-0-143-161.ec2.internal\" DevicePath \"\""
Sep 16 20:03:55.684093 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.684056    2578 generic.go:356] "Generic (PLEG): container finished" podID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerID="10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92" exitCode=0
Sep 16 20:03:55.684546 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.684103    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"
Sep 16 20:03:55.684546 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.684140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" event={"ID":"f8ce49b5-db92-4930-9f44-4f341b190bde","Type":"ContainerDied","Data":"10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92"}
Sep 16 20:03:55.684546 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.684203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj" event={"ID":"f8ce49b5-db92-4930-9f44-4f341b190bde","Type":"ContainerDied","Data":"2e441227091b867b34a9896931aff0d4cfd600c07a9d8d15a813367bca26261c"}
Sep 16 20:03:55.684546 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.684227    2578 scope.go:117] "RemoveContainer" containerID="10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92"
Sep 16 20:03:55.693948 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.693926    2578 scope.go:117] "RemoveContainer" containerID="a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f"
Sep 16 20:03:55.701802 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.701783    2578 scope.go:117] "RemoveContainer" containerID="10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92"
Sep 16 20:03:55.702106 ip-10-0-143-161 kubenswrapper[2578]: E0916 20:03:55.702079    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92\": container with ID starting with 10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92 not found: ID does not exist" containerID="10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92"
Sep 16 20:03:55.702194 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.702118    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92"} err="failed to get container status \"10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92\": rpc error: code = NotFound desc = could not find container \"10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92\": container with ID starting with 10c39eca049ca8182d8dba27d0b3d2eca95ecbeaf27ca03134e1df0cddb80d92 not found: ID does not exist"
Sep 16 20:03:55.702194 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.702143    2578 scope.go:117] "RemoveContainer" containerID="a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f"
Sep 16 20:03:55.702417 ip-10-0-143-161 kubenswrapper[2578]: E0916 20:03:55.702398    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f\": container with ID starting with a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f not found: ID does not exist" containerID="a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f"
Sep 16 20:03:55.702463 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.702424    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f"} err="failed to get container status \"a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f\": rpc error: code = NotFound desc = could not find container \"a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f\": container with ID starting with a45f772525ec56bc9158ae2e75b4e320bf24bcb5a5b2f9674333c3cebfcb3a1f not found: ID does not exist"
Sep 16 20:03:55.711289 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.711265    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"]
Sep 16 20:03:55.715753 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:55.715726    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5568cc8c6ddrjzj"]
Sep 16 20:03:57.063647 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:03:57.063611    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" path="/var/lib/kubelet/pods/f8ce49b5-db92-4930-9f44-4f341b190bde/volumes"
Sep 16 20:04:26.158933 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:26.158891    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df8cfc95d-tzm6f_78788e11-73c4-423d-ac3a-d2b71d6f70b8/router/0.log"
Sep 16 20:04:27.281844 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:27.281811    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df8cfc95d-tzm6f_78788e11-73c4-423d-ac3a-d2b71d6f70b8/router/0.log"
Sep 16 20:04:28.499827 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:28.499795    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-wrrqb_736deb6d-692f-4e80-b283-cfd74634c900/manager/0.log"
Sep 16 20:04:31.037575 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.037535    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-srgqk/must-gather-d8n77"]
Sep 16 20:04:31.038079 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.038054    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="storage-initializer"
Sep 16 20:04:31.038163 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.038083    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="storage-initializer"
Sep 16 20:04:31.038163 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.038101    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main"
Sep 16 20:04:31.038163 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.038109    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main"
Sep 16 20:04:31.038322 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.038198    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ce49b5-db92-4930-9f44-4f341b190bde" containerName="main"
Sep 16 20:04:31.041012 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.040988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srgqk/must-gather-d8n77"
Sep 16 20:04:31.045190 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.045165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-srgqk\"/\"default-dockercfg-tv7gj\""
Sep 16 20:04:31.046175 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.046154    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-srgqk\"/\"openshift-service-ca.crt\""
Sep 16 20:04:31.046288 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.046221    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-srgqk\"/\"kube-root-ca.crt\""
Sep 16 20:04:31.064754 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.064724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srgqk/must-gather-d8n77"]
Sep 16 20:04:31.154132 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.154093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d9c3f669-0cbb-46ba-8e4c-bca354726dcb-must-gather-output\") pod \"must-gather-d8n77\" (UID: \"d9c3f669-0cbb-46ba-8e4c-bca354726dcb\") " pod="openshift-must-gather-srgqk/must-gather-d8n77"
Sep 16 20:04:31.154132 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.154131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcgkk\" (UniqueName: \"kubernetes.io/projected/d9c3f669-0cbb-46ba-8e4c-bca354726dcb-kube-api-access-vcgkk\") pod \"must-gather-d8n77\" (UID: \"d9c3f669-0cbb-46ba-8e4c-bca354726dcb\") " pod="openshift-must-gather-srgqk/must-gather-d8n77"
Sep 16 20:04:31.255633 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.255575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d9c3f669-0cbb-46ba-8e4c-bca354726dcb-must-gather-output\") pod \"must-gather-d8n77\" (UID: \"d9c3f669-0cbb-46ba-8e4c-bca354726dcb\") " pod="openshift-must-gather-srgqk/must-gather-d8n77"
Sep 16 20:04:31.255633 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.255625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcgkk\" (UniqueName: \"kubernetes.io/projected/d9c3f669-0cbb-46ba-8e4c-bca354726dcb-kube-api-access-vcgkk\") pod \"must-gather-d8n77\" (UID: \"d9c3f669-0cbb-46ba-8e4c-bca354726dcb\") " pod="openshift-must-gather-srgqk/must-gather-d8n77"
Sep 16 20:04:31.255965 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.255944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d9c3f669-0cbb-46ba-8e4c-bca354726dcb-must-gather-output\") pod \"must-gather-d8n77\" (UID: \"d9c3f669-0cbb-46ba-8e4c-bca354726dcb\") " pod="openshift-must-gather-srgqk/must-gather-d8n77"
Sep 16 20:04:31.275799 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.275766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcgkk\" (UniqueName: \"kubernetes.io/projected/d9c3f669-0cbb-46ba-8e4c-bca354726dcb-kube-api-access-vcgkk\") pod \"must-gather-d8n77\" (UID: \"d9c3f669-0cbb-46ba-8e4c-bca354726dcb\") " pod="openshift-must-gather-srgqk/must-gather-d8n77"
Sep 16 20:04:31.349852 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.349766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srgqk/must-gather-d8n77"
Sep 16 20:04:31.627639 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.627545    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srgqk/must-gather-d8n77"]
Sep 16 20:04:31.631069 ip-10-0-143-161 kubenswrapper[2578]: W0916 20:04:31.631031    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9c3f669_0cbb_46ba_8e4c_bca354726dcb.slice/crio-ffad2744b849d37386933b6c368a2ca1bbeb3fc40ec7a75f2bf484a45ea4ef90 WatchSource:0}: Error finding container ffad2744b849d37386933b6c368a2ca1bbeb3fc40ec7a75f2bf484a45ea4ef90: Status 404 returned error can't find the container with id ffad2744b849d37386933b6c368a2ca1bbeb3fc40ec7a75f2bf484a45ea4ef90
Sep 16 20:04:31.632925 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.632905    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:04:31.854501 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:31.854444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srgqk/must-gather-d8n77" event={"ID":"d9c3f669-0cbb-46ba-8e4c-bca354726dcb","Type":"ContainerStarted","Data":"ffad2744b849d37386933b6c368a2ca1bbeb3fc40ec7a75f2bf484a45ea4ef90"}
Sep 16 20:04:32.860850 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:32.860809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srgqk/must-gather-d8n77" event={"ID":"d9c3f669-0cbb-46ba-8e4c-bca354726dcb","Type":"ContainerStarted","Data":"70ff5162e5f9b508ae900e32510d7a57202e254d1604a9e6360de3d8dea2cf10"}
Sep 16 20:04:32.861268 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:32.860864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srgqk/must-gather-d8n77" event={"ID":"d9c3f669-0cbb-46ba-8e4c-bca354726dcb","Type":"ContainerStarted","Data":"52f16cd5e0e18f20ebef41e0717cd07a691679d04260051455be19a3fb6b37d7"}
Sep 16 20:04:32.886863 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:32.886795    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-srgqk/must-gather-d8n77" podStartSLOduration=2.244523629 podStartE2EDuration="2.886771742s" totalImagesPullingTime="642.248113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:32.88613613 +0000 UTC m=+4780.333230660" watchObservedRunningTime="2026-09-16 20:04:32.886771742 +0000 UTC m=+4780.333866273"
Sep 16 20:04:34.391810 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:34.391770    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rfcc8_8ce0de65-1777-4f99-ae24-df43d786f881/global-pull-secret-syncer/0.log"
Sep 16 20:04:34.522784 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:34.522748    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zhnkf_70f46be3-8082-4906-8168-ca1dff840646/konnectivity-agent/0.log"
Sep 16 20:04:34.611499 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:34.611456    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-161.ec2.internal_240b210d213d06d9b2a5c1727b8ee1a0/haproxy/0.log"
Sep 16 20:04:39.373087 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:39.373050    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-wrrqb_736deb6d-692f-4e80-b283-cfd74634c900/manager/0.log"
Sep 16 20:04:41.882906 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:41.882233    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-zdc75_050e092a-8786-47a9-ab97-58c8b3952f06/monitoring-plugin/0.log"
Sep 16 20:04:41.966524 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:41.966456    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2dxr_5400a229-131f-4266-b766-b79aa5e298bc/node-exporter/0.log"
Sep 16 20:04:42.019343 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.019301    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2dxr_5400a229-131f-4266-b766-b79aa5e298bc/kube-rbac-proxy/0.log"
Sep 16 20:04:42.078107 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.078060    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2dxr_5400a229-131f-4266-b766-b79aa5e298bc/init-textfile/0.log"
Sep 16 20:04:42.420503 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.420456    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-rgdxh_b77a2558-789a-4ceb-a510-2bd83ec40efa/kube-rbac-proxy-main/0.log"
Sep 16 20:04:42.461564 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.461527    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-rgdxh_b77a2558-789a-4ceb-a510-2bd83ec40efa/kube-rbac-proxy-self/0.log"
Sep 16 20:04:42.508990 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.508946    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-rgdxh_b77a2558-789a-4ceb-a510-2bd83ec40efa/openshift-state-metrics/0.log"
Sep 16 20:04:42.586664 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.586629    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0db0dce-59ad-48d4-97fa-d97a54af1bce/prometheus/0.log"
Sep 16 20:04:42.628893 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.628864    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0db0dce-59ad-48d4-97fa-d97a54af1bce/config-reloader/0.log"
Sep 16 20:04:42.663679 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.663649    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0db0dce-59ad-48d4-97fa-d97a54af1bce/thanos-sidecar/0.log"
Sep 16 20:04:42.695708 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.695622    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0db0dce-59ad-48d4-97fa-d97a54af1bce/kube-rbac-proxy-web/0.log"
Sep 16 20:04:42.731649 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.731617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0db0dce-59ad-48d4-97fa-d97a54af1bce/kube-rbac-proxy/0.log"
Sep 16 20:04:42.779542 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.779511    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0db0dce-59ad-48d4-97fa-d97a54af1bce/kube-rbac-proxy-thanos/0.log"
Sep 16 20:04:42.823043 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.823012    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a0db0dce-59ad-48d4-97fa-d97a54af1bce/init-config-reloader/0.log"
Sep 16 20:04:42.867188 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.867153    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-w9kfm_49729b4e-8f6c-4b46-8cb0-8cae82929ab5/prometheus-operator/0.log"
Sep 16 20:04:42.891367 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.891336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-w9kfm_49729b4e-8f6c-4b46-8cb0-8cae82929ab5/kube-rbac-proxy/0.log"
Sep 16 20:04:42.927284 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:42.927243    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-zqmjb_cdcbfc31-d826-47fe-870e-3ca14c6ef6bc/prometheus-operator-admission-webhook/0.log"
Sep 16 20:04:43.076300 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:43.076269    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ddf59fb6d-btptq_796a1899-b668-4af7-a6b1-d90d63d3c998/thanos-query/0.log"
Sep 16 20:04:43.108020 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:43.107986    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ddf59fb6d-btptq_796a1899-b668-4af7-a6b1-d90d63d3c998/kube-rbac-proxy-web/0.log"
Sep 16 20:04:43.137537 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:43.137502    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ddf59fb6d-btptq_796a1899-b668-4af7-a6b1-d90d63d3c998/kube-rbac-proxy/0.log"
Sep 16 20:04:43.169638 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:43.169591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ddf59fb6d-btptq_796a1899-b668-4af7-a6b1-d90d63d3c998/prom-label-proxy/0.log"
Sep 16 20:04:43.211593 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:43.211545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ddf59fb6d-btptq_796a1899-b668-4af7-a6b1-d90d63d3c998/kube-rbac-proxy-rules/0.log"
Sep 16 20:04:43.238646 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:43.238613    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ddf59fb6d-btptq_796a1899-b668-4af7-a6b1-d90d63d3c998/kube-rbac-proxy-metrics/0.log"
Sep 16 20:04:43.991417 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:43.991381    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"]
Sep 16 20:04:43.997357 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:43.997324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.010341 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.010305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"]
Sep 16 20:04:44.082647 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.082601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-lib-modules\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.082859 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.082660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-podres\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.082859 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.082730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-proc\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.082859 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.082764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-sys\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.082859 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.082802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2qg\" (UniqueName: \"kubernetes.io/projected/6a128c9f-4931-46bf-9f0c-8999640140d8-kube-api-access-np2qg\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.183975 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.183937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np2qg\" (UniqueName: \"kubernetes.io/projected/6a128c9f-4931-46bf-9f0c-8999640140d8-kube-api-access-np2qg\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.184181 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.184088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-lib-modules\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.184181 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.184126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-podres\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.184300 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.184192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-proc\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.184300 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.184221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-sys\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.184410 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.184301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-sys\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.184410 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.184301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-lib-modules\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.184410 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.184360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-proc\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.184410 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.184372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a128c9f-4931-46bf-9f0c-8999640140d8-podres\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.193148 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.193119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2qg\" (UniqueName: \"kubernetes.io/projected/6a128c9f-4931-46bf-9f0c-8999640140d8-kube-api-access-np2qg\") pod \"perf-node-gather-daemonset-8ng59\" (UID: \"6a128c9f-4931-46bf-9f0c-8999640140d8\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.278692 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.278609    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-6k7dk_cc160e38-cf9b-49a1-be39-0afb8eaca6ac/networking-console-plugin/0.log"
Sep 16 20:04:44.313745 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.313706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.622968 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.622919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"]
Sep 16 20:04:44.625225 ip-10-0-143-161 kubenswrapper[2578]: W0916 20:04:44.625196    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a128c9f_4931_46bf_9f0c_8999640140d8.slice/crio-76e0ecb0348e27dc6d283df3cf6931daacbbb568013ebcf69bb0d09bca19519d WatchSource:0}: Error finding container 76e0ecb0348e27dc6d283df3cf6931daacbbb568013ebcf69bb0d09bca19519d: Status 404 returned error can't find the container with id 76e0ecb0348e27dc6d283df3cf6931daacbbb568013ebcf69bb0d09bca19519d
Sep 16 20:04:44.850366 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.850329    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 20:04:44.855882 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.855852    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/2.log"
Sep 16 20:04:44.950670 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.950574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59" event={"ID":"6a128c9f-4931-46bf-9f0c-8999640140d8","Type":"ContainerStarted","Data":"24eac55857f7cce3c6337c6526c80e26a34546cd3038bef7c381a7324ce1d6bd"}
Sep 16 20:04:44.950833 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.950689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:44.950833 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.950718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59" event={"ID":"6a128c9f-4931-46bf-9f0c-8999640140d8","Type":"ContainerStarted","Data":"76e0ecb0348e27dc6d283df3cf6931daacbbb568013ebcf69bb0d09bca19519d"}
Sep 16 20:04:44.968904 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:44.968849    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59" podStartSLOduration=1.9688318630000001 podStartE2EDuration="1.968831863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:44.968301831 +0000 UTC m=+4792.415396387" watchObservedRunningTime="2026-09-16 20:04:44.968831863 +0000 UTC m=+4792.415926442"
Sep 16 20:04:45.387962 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:45.387920    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-gl9hm_2ca91436-3300-447c-a199-5e12d7b3f511/download-server/0.log"
Sep 16 20:04:46.009694 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:46.009663    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-xgjxf_9761c3ca-de69-4601-98a8-bd173480628c/volume-data-source-validator/0.log"
Sep 16 20:04:47.128598 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:47.128563    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h7lbj_0e084045-524d-49ab-9191-7a4c3430cf0c/dns/0.log"
Sep 16 20:04:47.170656 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:47.170619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h7lbj_0e084045-524d-49ab-9191-7a4c3430cf0c/kube-rbac-proxy/0.log"
Sep 16 20:04:47.365281 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:47.365255    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9pgsd_b7f9ed7d-0ea5-4c26-b542-703da8f84edb/dns-node-resolver/0.log"
Sep 16 20:04:48.305859 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:48.305830    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zh85g_acb7ae1c-13d5-4745-aa4d-8e5f951e8f15/node-ca/0.log"
Sep 16 20:04:49.449751 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:49.449722    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df8cfc95d-tzm6f_78788e11-73c4-423d-ac3a-d2b71d6f70b8/router/0.log"
Sep 16 20:04:50.028760 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:50.028730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-djtj7_5fe145cc-75f4-4d0e-844e-4822424e1071/serve-healthcheck-canary/0.log"
Sep 16 20:04:50.701719 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:50.701689    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-67rrr_6afc2843-18bc-4f0f-a625-3a803fbf76cc/kube-rbac-proxy/0.log"
Sep 16 20:04:50.728739 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:50.728706    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-67rrr_6afc2843-18bc-4f0f-a625-3a803fbf76cc/exporter/0.log"
Sep 16 20:04:50.769127 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:50.769078    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-67rrr_6afc2843-18bc-4f0f-a625-3a803fbf76cc/extractor/0.log"
Sep 16 20:04:50.971274 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:50.971187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-8ng59"
Sep 16 20:04:53.442930 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:53.442899    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 20:04:53.450587 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:53.450551    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 20:04:53.469611 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:53.469581    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f6h98_391ce6ca-1cf6-458a-a904-59997c7eb6db/console-operator/1.log"
Sep 16 20:04:53.475501 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:53.475463    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knlxg_e943c3d7-af76-4889-9ab2-a272b6aef96a/ovn-acl-logging/0.log"
Sep 16 20:04:54.203508 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:54.203393    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df8cfc95d-tzm6f_78788e11-73c4-423d-ac3a-d2b71d6f70b8/router/0.log"
Sep 16 20:04:54.544176 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:54.544141    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-796cdc484c-94n74_9b9d9397-e72b-4637-9384-d5df58fc4c3a/manager/0.log"
Sep 16 20:04:54.602535 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:54.602493    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ksxq9_892f724b-9e97-4bf9-8072-2d3e690bbfd5/openshift-lws-operator/0.log"
Sep 16 20:04:55.351693 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:55.351638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-75c4c6c467-hb627_185e38f2-6449-4cda-ab93-632d097a2278/manager/0.log"
Sep 16 20:04:55.402104 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:55.402075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7df8cfc95d-tzm6f_78788e11-73c4-423d-ac3a-d2b71d6f70b8/router/0.log"
Sep 16 20:04:56.866661 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:56.866573    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-mp2wf_d5e1865e-3bfb-4e67-9cee-5d52d60812ce/manager/0.log"
Sep 16 20:04:56.945619 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:56.945583    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jx5ws_39d82901-ba40-4aed-a787-8d02b9a6db70/s3-init/0.log"
Sep 16 20:04:56.968381 ip-10-0-143-161 kubenswrapper[2578]: I0916 20:04:56.968349    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jx5ws_39d82901-ba40-4aed-a787-8d02b9a6db70/download-hf-model/0.log"