Sep 16 18:37:31.267514 ip-10-0-142-197 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:37:31.267525 ip-10-0-142-197 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:37:31.267533 ip-10-0-142-197 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:37:31.267710 ip-10-0-142-197 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:37:41.384296 ip-10-0-142-197 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:37:41.384314 ip-10-0-142-197 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d69afef6ad1d484fa5f61cc6478b20ab --
Sep 16 18:39:16.207123 ip-10-0-142-197 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:39:16.636002 ip-10-0-142-197 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:16.636002 ip-10-0-142-197 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:39:16.636002 ip-10-0-142-197 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:16.636002 ip-10-0-142-197 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:39:16.636002 ip-10-0-142-197 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:16.638508 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.638418    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:39:16.641808 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641780    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:16.641808 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641808    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641813    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641816    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641820    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641823    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641826    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641828    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641831    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641834    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641836    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641839    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641841    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641844    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641847    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641849    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641852    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641854    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641857    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641859    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641862    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:16.641872 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641864    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641867    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641869    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641880    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641883    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641886    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641888    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641891    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641893    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641896    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641898    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641901    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641904    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641907    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641910    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641912    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641915    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641918    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641921    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641924    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:16.642366 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641926    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641929    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641931    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641934    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641936    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641939    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641942    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641944    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641947    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641949    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641952    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641954    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641959    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641963    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641966    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641970    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641973    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641976    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641980    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:16.642881 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641984    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641987    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641990    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641992    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641995    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.641997    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642000    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642004    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642008    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642011    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642014    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642017    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642021    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642024    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642027    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642029    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642032    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642034    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642037    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642039    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:16.643358 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642042    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642044    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642046    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642049    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642051    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642054    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642057    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642060    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642062    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642065    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642067    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642070    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642073    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642075    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642077    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642080    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642083    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642086    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642088    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642807    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:16.643870 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642814    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642818    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642821    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642823    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642828    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642831    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642833    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642836    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642838    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642841    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642844    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642847    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642849    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642852    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642854    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642857    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642860    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642862    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642866    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642868    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:16.644386 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642871    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642874    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642877    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642881    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642885    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642888    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642891    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642893    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642896    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642899    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642901    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642905    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642909    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642912    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642915    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642918    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642921    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642924    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:16.644913 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642927    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642930    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642933    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642937    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642939    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642942    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642944    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642947    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642949    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642952    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642954    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642956    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642959    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642962    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642965    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642967    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642971    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642974    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642976    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642978    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:16.645365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642981    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642983    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642986    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642988    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642991    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642994    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642996    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.642999    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643002    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643004    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643007    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643009    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643013    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643016    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643019    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643021    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643024    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643026    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643029    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643031    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:16.645884 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643034    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643036    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643039    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643042    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643045    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643047    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643050    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643052    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643055    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643058    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643061    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643063    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643066    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643069    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643072    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643074    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643079    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643082    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643085    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643088    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:16.646392 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643510    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643516    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643520    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643523    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643526    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643529    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643533    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643536    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643539    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643542    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643545    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643550    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643553    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643556    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643559    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643562    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643565    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643567    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643570    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643572    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:16.646917 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643575    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643578    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643580    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643583    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643586    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643588    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643591    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643593    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643596    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643599    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643601    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643605    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643608    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643611    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643613    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643616    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643619    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643622    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643625    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643629    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:16.647425 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643632    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643635    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643638    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643642    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643645    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643648    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643664    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643669    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643672    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643675    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643678    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643681    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643683    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643686    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643689    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643692    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643694    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643697    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643700    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:16.647944 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643704    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643709    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643712    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643715    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643719    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643722    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643725    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643728    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643731    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643733    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643736    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643740    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643743    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643746    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643748    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643751    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643754    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643757    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643760    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:16.648501 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643763    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643765    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643768    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643770    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643773    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643775    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643778    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643781    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643783    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643798    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643801    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643804    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643806    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643809    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643811    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643814    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643817    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643821    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643823    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643826    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:16.649018 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.643828    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643895    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643935    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643943    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643947    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643957    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643961    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643965    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643970    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643975    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643978    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643982    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643985    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643989    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643992    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643994    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.643997    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644000    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644004    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644008    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644011    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644014    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644017    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644020    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:39:16.649544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644024    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644027    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644030    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644034    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644037    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644041    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644044    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644047    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644050    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644055    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644058    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644061    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644063    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644067    2561 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644070    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644075    2561 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644078    2561 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644082    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644085    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644088    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644092    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644095    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644098    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644100    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644103    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:39:16.650157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644106    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644110    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644113    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644116    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644118    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644121    2561 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644125    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644128    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644131    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644134    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644137    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644140    2561 flags.go:64] FLAG: --help="false"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644144    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-142-197.ec2.internal"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644148    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644150    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644153    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644157    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644160    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644163    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644166    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644169    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644171    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644175    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644178    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:39:16.650807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644183    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644186    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644189    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644192    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644195    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644198    2561 flags.go:64] FLAG: --lock-file=""
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644201    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644204    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644207    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644212    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644215    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644218    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644221    2561 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644224    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644227    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644230    2561 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644233    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644237    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644240    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644244    2561 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644249    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644252    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644255    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644258    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644261    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:39:16.651476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644264    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644267    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644274    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644277    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644280    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644283    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644287    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644292    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644297    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644300    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644303    2561 flags.go:64] FLAG: --port="10250"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644306    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644309    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0913d99b26a5400c3"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644312    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644315    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644318    2561 flags.go:64] FLAG: --register-node="true"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644321    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644324    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644327    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644330    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644333    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644336    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644339    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644342    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644345    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:39:16.652106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644348    2561 flags.go:64] FLAG: --runonce="false"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644351    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644359    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644362    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644365    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644368    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644371    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644374    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644377    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644380    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644383    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644386    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644389    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644392    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644396    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644400    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644428    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644432    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644435    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644439    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644442    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644445    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644448    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644451    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644453    2561 flags.go:64] FLAG: --v="2"
Sep 16 18:39:16.652733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644457    2561 flags.go:64] FLAG: --version="false"
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644462    2561 flags.go:64] FLAG: --vmodule=""
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644465    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644468    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644586    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644591    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644594    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644597    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644600    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644604    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644607    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644610    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644613    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644616    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644618    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644621    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644624    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644626    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644629    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644632    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:16.653403 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644634    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644637    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644641    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644643    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644646    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644649    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644651    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644654    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644657    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644659    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644662    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644665    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644667    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644670    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644673    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644677    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644680    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644683    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644686    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:16.653924 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644689    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644692    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644695    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644699    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644701    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644704    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644706    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644709    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644712    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644714    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644717    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644720    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644722    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644725    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644727    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644730    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644739    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644742    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644745    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644747    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:16.654414 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644750    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644752    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644755    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644757    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644760    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644772    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644775    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644778    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644780    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644783    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644785    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644803    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644808    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644811    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644816    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644819    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644821    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644824    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644826    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:16.654947 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644829    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644832    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644834    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644837    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644840    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644842    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644845    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644848    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644851    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644853    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644858    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644862    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644865    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644868    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644871    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644873    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644876    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644879    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644881    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644883    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:16.655431 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644886    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644888    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644891    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644893    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.644896    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.644902    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.652531    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.652548    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.653487    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.653495    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653552    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653557    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653560    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653563    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653566    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:16.655946 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653569    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653572    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653575    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653578    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653580    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653585    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653589    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653592    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653595    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653598    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653601    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653604    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653607    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653610    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653612    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653615    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653617    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653620    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653623    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:16.656328 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653625    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653628    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653630    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653633    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653636    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653640    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653644    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653647    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653649    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653652    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653655    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653658    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653660    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653663    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653666    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653669    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653672    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653674    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653677    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:16.656860 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653680    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653682    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653685    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653687    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653690    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653692    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653695    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653697    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653700    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653702    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653705    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653707    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653710    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653713    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653715    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653718    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653720    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653723    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653727    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653730    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653733    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:16.657338 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653736    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653738    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653741    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653743    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653747    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653750    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653754    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653757    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653759    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653762    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653766    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653770    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653773    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653776    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653779    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653782    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653784    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653799    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653803    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:16.657894 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653805    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653808    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653811    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653813    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653816    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653819    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653822    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653825    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653827    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653830    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653833    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653836    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653839    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653842    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653845    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:16.658365 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653847    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.653852    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653967    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653971    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653976    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653980    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653983    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653986    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653988    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653991    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653994    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.653997    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654000    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654002    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654005    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:16.658761 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654008    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654010    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654013    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654016    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654020    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654023    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654025    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654028    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654031    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654034    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654036    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654039    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654042    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654045    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654047    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654051    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654053    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654056    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654058    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:16.659153 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654061    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654064    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654066    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654069    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654071    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654074    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654076    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654079    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654081    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654084    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654086    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654089    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654092    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654094    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654096    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654099    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654102    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654104    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654108    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:16.659612 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654111    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654114    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654116    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654119    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654122    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654124    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654127    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654130    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654133    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654135    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654139    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654142    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654144    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654147    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654150    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654152    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654155    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654157    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654160    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654162    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:16.660112 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654165    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654167    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654170    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654172    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654175    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654177    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654180    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654182    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654185    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654188    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654190    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654193    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654195    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654198    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654200    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654203    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654206    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654208    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654211    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654214    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:16.660833 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654216    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654219    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654222    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654225    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654227    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654230    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654232    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:16.654235    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.654240    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.654366    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.657450    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.658587    2561 server.go:1019] "Starting client certificate rotation"
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.658689    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:16.661616 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.658726    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:16.664057 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.664037    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:16.666209 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.666190    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:16.676039 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.676017    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:39:16.679433 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.679418    2561 log.go:25] "Validated CRI v1 image API"
Sep 16 18:39:16.682495 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.682470    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:39:16.686774 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.686756    2561 fs.go:135] Filesystem UUIDs: map[5fb0e251-ba4f-41ce-abd8-785c24bbe418:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ad5f5b0a-e896-4f06-a320-7de795c072aa:/dev/nvme0n1p4]
Sep 16 18:39:16.686866 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.686774    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:39:16.691733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.691716    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:16.692264 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.692145    2561 manager.go:216] Machine: {Timestamp:2026-09-16 18:39:16.690499656 +0000 UTC m=+0.375605902 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095330 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec284d66c8b3792c74a97e1ddee4bc01 SystemUUID:ec284d66-c8b3-792c-74a9-7e1ddee4bc01 BootID:d69afef6-ad1d-484f-a5f6-1cc6478b20ab Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:b8:d6:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:b8:d6:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:39:f6:89:75:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:39:16.692264 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.692260    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:39:16.692364 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.692344    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:39:16.695707 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.695678    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:39:16.695878 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.695711    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-197.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:39:16.696581 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.696571    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:39:16.696612 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.696585    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:39:16.696612 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.696593    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:16.696612 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.696607    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:16.697896 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.697886    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:39:16.702653 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.702639    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:16.702813 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.702785    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:39:16.702874 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.702865    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:39:16.702916 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.702883    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:39:16.702916 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.702906    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:39:16.702993 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.702922    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:39:16.702993 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.702934    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:39:16.703369 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.703349    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:16.703488 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.703375    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:16.705749 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.705729    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:39:16.709768 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.709725    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:39:16.709877 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.709809    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:39:16.712568 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712553    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712572    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712578    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712583    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712592    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712597    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712610    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712615    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712622    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712628    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712636    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:39:16.712641 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.712641    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:39:16.713582 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.713571    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:39:16.713582 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.713581    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:39:16.715677 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.715659    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-197.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:39:16.715737 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.715675    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:39:16.715737 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.715675    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:39:16.717454 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.717436    2561 server.go:1268] "Started kubelet"
Sep 16 18:39:16.717535 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.717506    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:39:16.717627 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.717577    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:39:16.717714 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.717671    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:39:16.718217 ip-10-0-142-197 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:39:16.720307 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.720286    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:39:16.720605 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.720588    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:39:16.726399 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.726380    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:39:16.726527 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.725361    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-197.ec2.internal.18d5e12083e1060f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-197.ec2.internal,UID:ip-10-0-142-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-197.ec2.internal,},FirstTimestamp:2026-09-16 18:39:16.717405711 +0000 UTC m=+0.402511957,LastTimestamp:2026-09-16 18:39:16.717405711 +0000 UTC m=+0.402511957,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-197.ec2.internal,}"
Sep 16 18:39:16.732765 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.732749    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:16.733430 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.733402    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:39:16.733878 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.733850    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:16.734164 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734150    2561 factory.go:55] Registering systemd factory
Sep 16 18:39:16.734246 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734168    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:39:16.734246 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734175    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:39:16.734246 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734195    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:39:16.734246 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734176    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:39:16.734421 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734312    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:39:16.734421 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734320    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:39:16.734421 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734415    2561 factory.go:153] Registering CRI-O factory
Sep 16 18:39:16.734532 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734432    2561 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:39:16.734532 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734483    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:39:16.734532 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734499    2561 factory.go:103] Registering Raw factory
Sep 16 18:39:16.734532 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.734511    2561 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:39:16.735526 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.735495    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:39:16.735610 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.735512    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:39:16.735660 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.735612    2561 manager.go:349] Starting recovery of all containers
Sep 16 18:39:16.741811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.741603    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:39:16.746451 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.746433    2561 manager.go:354] Recovery completed
Sep 16 18:39:16.752684 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.752668    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:16.754553 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.754533    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:16.754644 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.754567    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:16.754644 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.754583    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:16.755202 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.755187    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:39:16.755202 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.755199    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:39:16.755295 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.755213    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:16.755295 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.755189    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:16.755884 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.755867    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:16.755949 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.755894    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:16.755949 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.755908    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:16.757376 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.757364    2561 policy_none.go:49] "None policy: Start"
Sep 16 18:39:16.757417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.757380    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:39:16.757417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.757390    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:39:16.757819 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.757805    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rxsvw"
Sep 16 18:39:16.759310 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.759300    2561 policy_none.go:47] "Start"
Sep 16 18:39:16.764643 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.764626    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rxsvw"
Sep 16 18:39:16.765081 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.765014    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-197.ec2.internal.18d5e1208617d1d4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-197.ec2.internal,UID:ip-10-0-142-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-197.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-197.ec2.internal,},FirstTimestamp:2026-09-16 18:39:16.754551252 +0000 UTC m=+0.439657507,LastTimestamp:2026-09-16 18:39:16.754551252 +0000 UTC m=+0.439657507,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-197.ec2.internal,}"
Sep 16 18:39:16.834474 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.834446    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:16.846124 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.846097    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:39:16.846246 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.846149    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:39:16.846246 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.846174    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:39:16.846246 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.846211    2561 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 16 18:39:16.848674 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.848654    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:16.877328 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.877313    2561 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:39:16.877414 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.877353    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:39:16.877414 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.877363    2561 server.go:87] "Starting device plugin registration server"
Sep 16 18:39:16.877738 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.877725    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:39:16.877784 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.877738    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:39:16.877907 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.877886    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:39:16.878016 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.877976    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:39:16.878016 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.877986    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:39:16.878617 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.878594    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:39:16.878720 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.878644    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:16.941574 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.941470    2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-197.ec2.internal\" not found" node="ip-10-0-142-197.ec2.internal"
Sep 16 18:39:16.946620 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.946585    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"]
Sep 16 18:39:16.946750 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.946660    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:16.948164 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.948147    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:16.948238 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.948176    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:16.948238 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.948188    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:16.950483 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.950467    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:16.950619 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.950603    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:16.950655 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.950637    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:16.951333 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.951313    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:16.951333 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.951323    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:16.951484 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.951345    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:16.951484 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.951356    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:16.951484 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.951359    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:16.951484 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.951372    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:16.953509 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.953492    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:16.953618 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.953520    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:16.954733 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.954716    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:16.954853 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.954742    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:16.954853 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.954751    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:16.978257 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.978225    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:16.979283 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.979263    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:16.979400 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.979301    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:16.979400 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.979315    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:16.979400 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.979339    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-197.ec2.internal"
Sep 16 18:39:16.979937 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.979919    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-197.ec2.internal\" not found" node="ip-10-0-142-197.ec2.internal"
Sep 16 18:39:16.984780 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.984763    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-197.ec2.internal\" not found" node="ip-10-0-142-197.ec2.internal"
Sep 16 18:39:16.987962 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:16.987945    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-197.ec2.internal"
Sep 16 18:39:16.988019 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:16.987973    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-197.ec2.internal\": node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:17.036035 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:17.035999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb15a171a0a140f446df999ca2a1dbe0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"fb15a171a0a140f446df999ca2a1dbe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:17.036035 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:17.036034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb15a171a0a140f446df999ca2a1dbe0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"fb15a171a0a140f446df999ca2a1dbe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:17.036211 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:17.036057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa12150102fe3893c84359d7f70bc6ce-config\") pod \"kube-apiserver-proxy-ip-10-0-142-197.ec2.internal\" (UID: \"aa12150102fe3893c84359d7f70bc6ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:17.049417 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:17.049395    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:17.136759 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:17.136727    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa12150102fe3893c84359d7f70bc6ce-config\") pod \"kube-apiserver-proxy-ip-10-0-142-197.ec2.internal\" (UID: \"aa12150102fe3893c84359d7f70bc6ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:17.136759 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:17.136759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb15a171a0a140f446df999ca2a1dbe0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"fb15a171a0a140f446df999ca2a1dbe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:17.136938 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:17.136775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb15a171a0a140f446df999ca2a1dbe0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"fb15a171a0a140f446df999ca2a1dbe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:17.136938 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:17.136859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa12150102fe3893c84359d7f70bc6ce-config\") pod \"kube-apiserver-proxy-ip-10-0-142-197.ec2.internal\" (UID: \"aa12150102fe3893c84359d7f70bc6ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:17.136938 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:17.136872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb15a171a0a140f446df999ca2a1dbe0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"fb15a171a0a140f446df999ca2a1dbe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:17.136938 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:17.136901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb15a171a0a140f446df999ca2a1dbe0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"fb15a171a0a140f446df999ca2a1dbe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:17.149870 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:17.149843    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:17.250635 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:17.250565    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:19.179614 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.179582    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:19.184673 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.184328    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:19.248356 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:19.248300    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:19.348808 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:19.348777    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:19.449415 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:19.449349    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:19.550022 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:19.549991    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:19.555160 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.555141    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:39:19.555280 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.555263    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:39:19.650379 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:19.650349    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:19.667523 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.667489    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:34:18 +0000 UTC" deadline="2028-04-11 01:29:19.677752553 +0000 UTC"
Sep 16 18:39:19.667523 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.667515    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13734h50m0.010240416s"
Sep 16 18:39:19.751403 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:19.751335    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:19.802772 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:19.802730    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa12150102fe3893c84359d7f70bc6ce.slice/crio-43d0a2092f7926e60261795f32c1195949af9c87e72ee3dc3e5c15d1f4d6653f WatchSource:0}: Error finding container 43d0a2092f7926e60261795f32c1195949af9c87e72ee3dc3e5c15d1f4d6653f: Status 404 returned error can't find the container with id 43d0a2092f7926e60261795f32c1195949af9c87e72ee3dc3e5c15d1f4d6653f
Sep 16 18:39:19.804531 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:19.804180    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb15a171a0a140f446df999ca2a1dbe0.slice/crio-8b313b7112d5fbe65d5d4edafd398964d7d51ee6607a21c0b59302d8bb066b69 WatchSource:0}: Error finding container 8b313b7112d5fbe65d5d4edafd398964d7d51ee6607a21c0b59302d8bb066b69: Status 404 returned error can't find the container with id 8b313b7112d5fbe65d5d4edafd398964d7d51ee6607a21c0b59302d8bb066b69
Sep 16 18:39:19.808634 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.808618    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:39:19.818266 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.818244    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:19.819172 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.819155    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:19.819279 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.819185    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:19.819279 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.819196    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:19.851847 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:19.851803    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:19.894157 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:19.894127    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:19.952828 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:19.952794    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 16 18:39:20.036519 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.036452    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:20.131268 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.131238    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:20.147037 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.147010    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:20.148066 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.148051    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 16 18:39:20.159076 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.159050    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:20.221407 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.221378    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:20.600988 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.600943    2561 apiserver.go:52] "Watching apiserver"
Sep 16 18:39:20.610626 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.610590    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:39:20.612706 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.612673    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx","openshift-cluster-node-tuning-operator/tuned-glc54","openshift-dns/node-resolver-rrwk9","openshift-multus/multus-additional-cni-plugins-khznh","openshift-multus/multus-n9wqg","openshift-multus/network-metrics-daemon-ds74d","kube-system/global-pull-secret-syncer-ztbzx","openshift-image-registry/node-ca-smzkz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal","openshift-network-diagnostics/network-check-target-2xjm7","openshift-network-operator/iptables-alerter-dqm5k","openshift-ovn-kubernetes/ovnkube-node-x7hhj","kube-system/konnectivity-agent-pvshm","kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"]
Sep 16 18:39:20.615263 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.615230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:20.615361 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.615323    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:20.617366 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.617341    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:20.617463 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.617416    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:20.619516 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.619484    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.622844 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.621828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.622844 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.622557    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:39:20.622844 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.622606    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:39:20.622844 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.622654    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:39:20.622844 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.622674    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-whsh9\""
Sep 16 18:39:20.622844 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.622579    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:39:20.622844 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.622816    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:39:20.624186 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.624155    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.625536 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.625520    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:39:20.626919 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.626307    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:39:20.626919 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.626419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p4cc5\""
Sep 16 18:39:20.626919 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.626443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f2jbf\""
Sep 16 18:39:20.626919 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.626483    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:39:20.626919 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.626624    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:39:20.626919 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.626640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.628979 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.628958    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5nvj7\""
Sep 16 18:39:20.629071 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.629005    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:39:20.629071 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.628958    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:39:20.629349 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.629330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:20.629452 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.629403    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:20.630267 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.630246    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:20.631690 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.631631    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.633871 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.633847    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.636740 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.636720    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:39:20.637184 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.636942    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:39:20.637184 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.637000    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jtwtb\""
Sep 16 18:39:20.637184 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.637026    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:39:20.637184 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.637067    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:20.637184 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.637153    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:20.637468 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.637256    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6jwkz\""
Sep 16 18:39:20.637468 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.637298    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.639315 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.639292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.639682 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.639610    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:39:20.640031 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.640011    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:20.640405 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.640124    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-prqfq\""
Sep 16 18:39:20.640405 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.640193    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:20.641483 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.641465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:20.641804 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.641672    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:39:20.641804 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.641737    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:39:20.641955 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.641895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6rjpw\""
Sep 16 18:39:20.642417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.642271    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:39:20.642417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.642373    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643035    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643517    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-modprobe-d\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-sysctl-conf\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0930c4f1-458b-47e4-a122-d1080055cd14-multus-daemon-config\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-sys-fs\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-device-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-sysconfig\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-cnibin\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-var-lib-kubelet\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.643975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2447bda6-429c-48a3-ad0a-e8b58294e883-kubelet-config\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wmfq\" (UniqueName: \"kubernetes.io/projected/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-kube-api-access-9wmfq\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644032    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:20.644492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-system-cni-dir\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0930c4f1-458b-47e4-a122-d1080055cd14-cni-binary-copy\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-var-lib-cni-bin\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrv4r\" (UniqueName: \"kubernetes.io/projected/0930c4f1-458b-47e4-a122-d1080055cd14-kube-api-access-mrv4r\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/46d448d3-6c08-4ec6-9d5d-c62c2635e0c2-hosts-file\") pod \"node-resolver-rrwk9\" (UID: \"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2\") " pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnjj9\" (UniqueName: \"kubernetes.io/projected/46d448d3-6c08-4ec6-9d5d-c62c2635e0c2-kube-api-access-cnjj9\") pod \"node-resolver-rrwk9\" (UID: \"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2\") " pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-run-netns\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-var-lib-cni-multus\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dda38eb1-7c5b-42cb-8557-640253d94a7f-system-cni-dir\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dda38eb1-7c5b-42cb-8557-640253d94a7f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-run-multus-certs\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dda38eb1-7c5b-42cb-8557-640253d94a7f-cnibin\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d448d3-6c08-4ec6-9d5d-c62c2635e0c2-tmp-dir\") pod \"node-resolver-rrwk9\" (UID: \"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2\") " pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-sysctl-d\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-sys\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.645385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644741    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dda38eb1-7c5b-42cb-8557-640253d94a7f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-multus-cni-dir\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-hostroot\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-multus-conf-dir\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644797    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-lib-modules\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644951    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-var-lib-kubelet\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-675vw\" (UniqueName: \"kubernetes.io/projected/960f29d3-0e56-4ca5-9f0a-d319b2107b04-kube-api-access-675vw\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-run-k8s-cni-cncf-io\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.644997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61c92186-ba53-4d0d-8c30-5dd1e3778b30-host\") pod \"node-ca-smzkz\" (UID: \"61c92186-ba53-4d0d-8c30-5dd1e3778b30\") " pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.645991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.645011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-socket-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.646731 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.645026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dda38eb1-7c5b-42cb-8557-640253d94a7f-cni-binary-copy\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.646833 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.646775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8pq6\" (UniqueName: \"kubernetes.io/projected/dda38eb1-7c5b-42cb-8557-640253d94a7f-kube-api-access-n8pq6\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.646963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.646845    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:39:20.646963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.646897    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f94h2\""
Sep 16 18:39:20.647547 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.647528    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:20.648051 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-kubernetes\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.648151 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-host\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.648206 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-multus-socket-dir-parent\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.648257 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km85r\" (UniqueName: \"kubernetes.io/projected/61c92186-ba53-4d0d-8c30-5dd1e3778b30-kube-api-access-km85r\") pod \"node-ca-smzkz\" (UID: \"61c92186-ba53-4d0d-8c30-5dd1e3778b30\") " pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.648413 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2447bda6-429c-48a3-ad0a-e8b58294e883-dbus\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:20.648464 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-run\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.648508 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/960f29d3-0e56-4ca5-9f0a-d319b2107b04-tmp\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.648547 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-etc-kubernetes\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.649140 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9bhp\" (UniqueName: \"kubernetes.io/projected/e08562f6-72fa-408c-9d9f-43ce0d4ced85-kube-api-access-k9bhp\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:20.649140 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dda38eb1-7c5b-42cb-8557-640253d94a7f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.649140 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61c92186-ba53-4d0d-8c30-5dd1e3778b30-serviceca\") pod \"node-ca-smzkz\" (UID: \"61c92186-ba53-4d0d-8c30-5dd1e3778b30\") " pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.649140 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-registration-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.649140 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-systemd\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.649140 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.648931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-tuned\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.650232 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.650206    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:20.678451 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.678425    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mdf8b"
Sep 16 18:39:20.685291 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.685269    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mdf8b"
Sep 16 18:39:20.732108 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.732073    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:39:20.748656 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.748598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal" event={"ID":"fb15a171a0a140f446df999ca2a1dbe0","Type":"ContainerStarted","Data":"8b313b7112d5fbe65d5d4edafd398964d7d51ee6607a21c0b59302d8bb066b69"}
Sep 16 18:39:20.749214 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-socket-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.749362 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-run-netns\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.749362 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dda38eb1-7c5b-42cb-8557-640253d94a7f-cni-binary-copy\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.749362 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8pq6\" (UniqueName: \"kubernetes.io/projected/dda38eb1-7c5b-42cb-8557-640253d94a7f-kube-api-access-n8pq6\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.749362 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-etc-openvswitch\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.749362 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa398820-2005-47bc-8687-b89d132bb6fb-konnectivity-ca\") pod \"konnectivity-agent-pvshm\" (UID: \"fa398820-2005-47bc-8687-b89d132bb6fb\") " pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-kubernetes\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749878    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-socket-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-host\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dda38eb1-7c5b-42cb-8557-640253d94a7f-cni-binary-copy\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-multus-socket-dir-parent\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-multus-socket-dir-parent\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.749990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km85r\" (UniqueName: \"kubernetes.io/projected/61c92186-ba53-4d0d-8c30-5dd1e3778b30-kube-api-access-km85r\") pod \"node-ca-smzkz\" (UID: \"61c92186-ba53-4d0d-8c30-5dd1e3778b30\") " pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-kubernetes\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-cni-bin\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-host\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-ovnkube-script-lib\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750058    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal" event={"ID":"aa12150102fe3893c84359d7f70bc6ce","Type":"ContainerStarted","Data":"43d0a2092f7926e60261795f32c1195949af9c87e72ee3dc3e5c15d1f4d6653f"}
Sep 16 18:39:20.750065 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2447bda6-429c-48a3-ad0a-e8b58294e883-dbus\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-run\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/960f29d3-0e56-4ca5-9f0a-d319b2107b04-tmp\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-etc-kubernetes\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9bhp\" (UniqueName: \"kubernetes.io/projected/e08562f6-72fa-408c-9d9f-43ce0d4ced85-kube-api-access-k9bhp\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dda38eb1-7c5b-42cb-8557-640253d94a7f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2447bda6-429c-48a3-ad0a-e8b58294e883-dbus\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-run-systemd\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-run\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-node-log\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-etc-kubernetes\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61c92186-ba53-4d0d-8c30-5dd1e3778b30-serviceca\") pod \"node-ca-smzkz\" (UID: \"61c92186-ba53-4d0d-8c30-5dd1e3778b30\") " pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-registration-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68b5df9d-b26c-4616-abc5-69b774ea6ea2-host-slash\") pod \"iptables-alerter-dqm5k\" (UID: \"68b5df9d-b26c-4616-abc5-69b774ea6ea2\") " pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-var-lib-openvswitch\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-ovn-node-metrics-cert\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dda38eb1-7c5b-42cb-8557-640253d94a7f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.750757 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-registration-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-systemd\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-tuned\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttt7l\" (UniqueName: \"kubernetes.io/projected/68b5df9d-b26c-4616-abc5-69b774ea6ea2-kube-api-access-ttt7l\") pod \"iptables-alerter-dqm5k\" (UID: \"68b5df9d-b26c-4616-abc5-69b774ea6ea2\") " pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-systemd\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-modprobe-d\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-sysctl-conf\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0930c4f1-458b-47e4-a122-d1080055cd14-multus-daemon-config\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-sys-fs\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-modprobe-d\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-device-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-sys-fs\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-sysconfig\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-sysctl-conf\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.751560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-cnibin\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.750784    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61c92186-ba53-4d0d-8c30-5dd1e3778b30-serviceca\") pod \"node-ca-smzkz\" (UID: \"61c92186-ba53-4d0d-8c30-5dd1e3778b30\") " pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-var-lib-kubelet\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-cnibin\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-var-lib-kubelet\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750880    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.750883    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs podName:e08562f6-72fa-408c-9d9f-43ce0d4ced85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:21.250842968 +0000 UTC m=+3.039171329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs") pod "network-metrics-daemon-ds74d" (UID: "e08562f6-72fa-408c-9d9f-43ce0d4ced85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-sysconfig\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2447bda6-429c-48a3-ad0a-e8b58294e883-kubelet-config\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750925    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750945    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wmfq\" (UniqueName: \"kubernetes.io/projected/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-kube-api-access-9wmfq\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750969    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-device-dir\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.750994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-log-socket\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.752458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-system-cni-dir\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2447bda6-429c-48a3-ad0a-e8b58294e883-kubelet-config\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0930c4f1-458b-47e4-a122-d1080055cd14-cni-binary-copy\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-var-lib-cni-bin\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrv4r\" (UniqueName: \"kubernetes.io/projected/0930c4f1-458b-47e4-a122-d1080055cd14-kube-api-access-mrv4r\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/46d448d3-6c08-4ec6-9d5d-c62c2635e0c2-hosts-file\") pod \"node-resolver-rrwk9\" (UID: \"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2\") " pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnjj9\" (UniqueName: \"kubernetes.io/projected/46d448d3-6c08-4ec6-9d5d-c62c2635e0c2-kube-api-access-cnjj9\") pod \"node-resolver-rrwk9\" (UID: \"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2\") " pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-systemd-units\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-system-cni-dir\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftq5\" (UniqueName: \"kubernetes.io/projected/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-kube-api-access-pftq5\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-run-netns\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-var-lib-cni-multus\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dda38eb1-7c5b-42cb-8557-640253d94a7f-system-cni-dir\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dda38eb1-7c5b-42cb-8557-640253d94a7f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-var-lib-cni-bin\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-run-ovn\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.753048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-run-netns\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-run-multus-certs\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-var-lib-cni-multus\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0930c4f1-458b-47e4-a122-d1080055cd14-cni-binary-copy\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.751534    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0930c4f1-458b-47e4-a122-d1080055cd14-multus-daemon-config\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/46d448d3-6c08-4ec6-9d5d-c62c2635e0c2-hosts-file\") pod \"node-resolver-rrwk9\" (UID: \"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2\") " pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dda38eb1-7c5b-42cb-8557-640253d94a7f-system-cni-dir\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.751612    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret podName:2447bda6-429c-48a3-ad0a-e8b58294e883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:21.251594092 +0000 UTC m=+3.039922439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret") pod "global-pull-secret-syncer-ztbzx" (UID: "2447bda6-429c-48a3-ad0a-e8b58294e883") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dda38eb1-7c5b-42cb-8557-640253d94a7f-cnibin\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dda38eb1-7c5b-42cb-8557-640253d94a7f-cnibin\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-run-multus-certs\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d448d3-6c08-4ec6-9d5d-c62c2635e0c2-tmp-dir\") pod \"node-resolver-rrwk9\" (UID: \"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2\") " pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68b5df9d-b26c-4616-abc5-69b774ea6ea2-iptables-alerter-script\") pod \"iptables-alerter-dqm5k\" (UID: \"68b5df9d-b26c-4616-abc5-69b774ea6ea2\") " pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa398820-2005-47bc-8687-b89d132bb6fb-agent-certs\") pod \"konnectivity-agent-pvshm\" (UID: \"fa398820-2005-47bc-8687-b89d132bb6fb\") " pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-sysctl-d\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.753588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-sys\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751832    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dda38eb1-7c5b-42cb-8557-640253d94a7f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-kubelet\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751880    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-multus-cni-dir\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-hostroot\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-multus-conf-dir\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-slash\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.751954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d448d3-6c08-4ec6-9d5d-c62c2635e0c2-tmp-dir\") pod \"node-resolver-rrwk9\" (UID: \"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2\") " pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752015    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-multus-cni-dir\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dda38eb1-7c5b-42cb-8557-640253d94a7f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-run-openvswitch\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-cni-netd\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-hostroot\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752100    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-ovnkube-config\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-multus-conf-dir\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752121    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-env-overrides\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-lib-modules\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.754250 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-sys\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-sysctl-d\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-var-lib-kubelet\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-675vw\" (UniqueName: \"kubernetes.io/projected/960f29d3-0e56-4ca5-9f0a-d319b2107b04-kube-api-access-675vw\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-var-lib-kubelet\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-run-k8s-cni-cncf-io\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61c92186-ba53-4d0d-8c30-5dd1e3778b30-host\") pod \"node-ca-smzkz\" (UID: \"61c92186-ba53-4d0d-8c30-5dd1e3778b30\") " pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/960f29d3-0e56-4ca5-9f0a-d319b2107b04-lib-modules\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61c92186-ba53-4d0d-8c30-5dd1e3778b30-host\") pod \"node-ca-smzkz\" (UID: \"61c92186-ba53-4d0d-8c30-5dd1e3778b30\") " pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0930c4f1-458b-47e4-a122-d1080055cd14-host-run-k8s-cni-cncf-io\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.754932 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.752392    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dda38eb1-7c5b-42cb-8557-640253d94a7f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.755475 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.755453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/960f29d3-0e56-4ca5-9f0a-d319b2107b04-tmp\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.755586 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.755485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/960f29d3-0e56-4ca5-9f0a-d319b2107b04-etc-tuned\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.760950 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.760887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8pq6\" (UniqueName: \"kubernetes.io/projected/dda38eb1-7c5b-42cb-8557-640253d94a7f-kube-api-access-n8pq6\") pod \"multus-additional-cni-plugins-khznh\" (UID: \"dda38eb1-7c5b-42cb-8557-640253d94a7f\") " pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.761070 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.761032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9bhp\" (UniqueName: \"kubernetes.io/projected/e08562f6-72fa-408c-9d9f-43ce0d4ced85-kube-api-access-k9bhp\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:20.761283 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.761260    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:20.761352 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.761287    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:20.761352 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.761303    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kw8st for pod openshift-network-diagnostics/network-check-target-2xjm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:20.761468 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:20.761376    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st podName:43c61e8e-f211-4b63-afc1-99c5c0571682 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:21.261358889 +0000 UTC m=+3.049687251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kw8st" (UniqueName: "kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st") pod "network-check-target-2xjm7" (UID: "43c61e8e-f211-4b63-afc1-99c5c0571682") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:20.761536 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.761516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-675vw\" (UniqueName: \"kubernetes.io/projected/960f29d3-0e56-4ca5-9f0a-d319b2107b04-kube-api-access-675vw\") pod \"tuned-glc54\" (UID: \"960f29d3-0e56-4ca5-9f0a-d319b2107b04\") " pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.762242 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.762199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrv4r\" (UniqueName: \"kubernetes.io/projected/0930c4f1-458b-47e4-a122-d1080055cd14-kube-api-access-mrv4r\") pod \"multus-n9wqg\" (UID: \"0930c4f1-458b-47e4-a122-d1080055cd14\") " pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.762242 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.762198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km85r\" (UniqueName: \"kubernetes.io/projected/61c92186-ba53-4d0d-8c30-5dd1e3778b30-kube-api-access-km85r\") pod \"node-ca-smzkz\" (UID: \"61c92186-ba53-4d0d-8c30-5dd1e3778b30\") " pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.762473 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.762453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnjj9\" (UniqueName: \"kubernetes.io/projected/46d448d3-6c08-4ec6-9d5d-c62c2635e0c2-kube-api-access-cnjj9\") pod \"node-resolver-rrwk9\" (UID: \"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2\") " pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.762987 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.762965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wmfq\" (UniqueName: \"kubernetes.io/projected/a206f1c7-dc3a-485e-8e7f-ddd6e09108f9-kube-api-access-9wmfq\") pod \"aws-ebs-csi-driver-node-xkpgx\" (UID: \"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.852750 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-systemd-units\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.852750 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftq5\" (UniqueName: \"kubernetes.io/projected/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-kube-api-access-pftq5\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.852750 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852736    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-systemd-units\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-run-ovn\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-run-ovn\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68b5df9d-b26c-4616-abc5-69b774ea6ea2-iptables-alerter-script\") pod \"iptables-alerter-dqm5k\" (UID: \"68b5df9d-b26c-4616-abc5-69b774ea6ea2\") " pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852859    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa398820-2005-47bc-8687-b89d132bb6fb-agent-certs\") pod \"konnectivity-agent-pvshm\" (UID: \"fa398820-2005-47bc-8687-b89d132bb6fb\") " pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-kubelet\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-slash\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-run-openvswitch\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.852982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-run-openvswitch\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853020 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-slash\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-kubelet\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-cni-netd\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-cni-netd\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-ovnkube-config\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-env-overrides\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-run-netns\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-etc-openvswitch\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa398820-2005-47bc-8687-b89d132bb6fb-konnectivity-ca\") pod \"konnectivity-agent-pvshm\" (UID: \"fa398820-2005-47bc-8687-b89d132bb6fb\") " pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-cni-bin\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-run-netns\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-ovnkube-script-lib\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-run-systemd\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-node-log\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68b5df9d-b26c-4616-abc5-69b774ea6ea2-host-slash\") pod \"iptables-alerter-dqm5k\" (UID: \"68b5df9d-b26c-4616-abc5-69b774ea6ea2\") " pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-var-lib-openvswitch\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-ovn-node-metrics-cert\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.853530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttt7l\" (UniqueName: \"kubernetes.io/projected/68b5df9d-b26c-4616-abc5-69b774ea6ea2-kube-api-access-ttt7l\") pod \"iptables-alerter-dqm5k\" (UID: \"68b5df9d-b26c-4616-abc5-69b774ea6ea2\") " pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-etc-openvswitch\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-log-socket\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68b5df9d-b26c-4616-abc5-69b774ea6ea2-iptables-alerter-script\") pod \"iptables-alerter-dqm5k\" (UID: \"68b5df9d-b26c-4616-abc5-69b774ea6ea2\") " pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68b5df9d-b26c-4616-abc5-69b774ea6ea2-host-slash\") pod \"iptables-alerter-dqm5k\" (UID: \"68b5df9d-b26c-4616-abc5-69b774ea6ea2\") " pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-log-socket\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-cni-bin\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-var-lib-openvswitch\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-env-overrides\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-run-systemd\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-ovnkube-config\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-node-log\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.853793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa398820-2005-47bc-8687-b89d132bb6fb-konnectivity-ca\") pod \"konnectivity-agent-pvshm\" (UID: \"fa398820-2005-47bc-8687-b89d132bb6fb\") " pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:20.854340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.854136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-ovnkube-script-lib\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.855885 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.855863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa398820-2005-47bc-8687-b89d132bb6fb-agent-certs\") pod \"konnectivity-agent-pvshm\" (UID: \"fa398820-2005-47bc-8687-b89d132bb6fb\") " pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:20.856003 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.855913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-ovn-node-metrics-cert\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.867606 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.867559    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftq5\" (UniqueName: \"kubernetes.io/projected/fa975d83-17d8-4510-8ba1-0b0bf700f6bb-kube-api-access-pftq5\") pod \"ovnkube-node-x7hhj\" (UID: \"fa975d83-17d8-4510-8ba1-0b0bf700f6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.867904 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.867879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttt7l\" (UniqueName: \"kubernetes.io/projected/68b5df9d-b26c-4616-abc5-69b774ea6ea2-kube-api-access-ttt7l\") pod \"iptables-alerter-dqm5k\" (UID: \"68b5df9d-b26c-4616-abc5-69b774ea6ea2\") " pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.939923 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.939886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-khznh"
Sep 16 18:39:20.949022 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.948999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx"
Sep 16 18:39:20.958891 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.958867    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9wqg"
Sep 16 18:39:20.962711 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.962692    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rrwk9"
Sep 16 18:39:20.971901 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.971881    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-smzkz"
Sep 16 18:39:20.977776 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.977757    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-glc54"
Sep 16 18:39:20.986322 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.986297    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dqm5k"
Sep 16 18:39:20.993956 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.993931    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:20.999596 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:20.999548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:21.256148 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.256060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:21.256148 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.256129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:21.256346 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:21.256236    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:21.256346 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:21.256307    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret podName:2447bda6-429c-48a3-ad0a-e8b58294e883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:22.256287421 +0000 UTC m=+4.044615767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret") pod "global-pull-secret-syncer-ztbzx" (UID: "2447bda6-429c-48a3-ad0a-e8b58294e883") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:21.256346 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:21.256239    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:21.256504 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:21.256391    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs podName:e08562f6-72fa-408c-9d9f-43ce0d4ced85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:22.256373289 +0000 UTC m=+4.044701647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs") pod "network-metrics-daemon-ds74d" (UID: "e08562f6-72fa-408c-9d9f-43ce0d4ced85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:21.356854 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.356818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:21.357032 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:21.356993    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:21.357032 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:21.357025    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:21.357105 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:21.357040    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kw8st for pod openshift-network-diagnostics/network-check-target-2xjm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:21.357105 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:21.357097    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st podName:43c61e8e-f211-4b63-afc1-99c5c0571682 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:22.357082629 +0000 UTC m=+4.145410978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kw8st" (UniqueName: "kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st") pod "network-check-target-2xjm7" (UID: "43c61e8e-f211-4b63-afc1-99c5c0571682") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:21.677442 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:21.677267    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda38eb1_7c5b_42cb_8557_640253d94a7f.slice/crio-a2fbd4bac525680718166221282454f7cdb6256c90467e3a894f0e6f2f3768db WatchSource:0}: Error finding container a2fbd4bac525680718166221282454f7cdb6256c90467e3a894f0e6f2f3768db: Status 404 returned error can't find the container with id a2fbd4bac525680718166221282454f7cdb6256c90467e3a894f0e6f2f3768db
Sep 16 18:39:21.679407 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:21.679370    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d448d3_6c08_4ec6_9d5d_c62c2635e0c2.slice/crio-f6a5970cb6a8d63cbcf72b226015a24f279bda652783dede713bd9138e3495f9 WatchSource:0}: Error finding container f6a5970cb6a8d63cbcf72b226015a24f279bda652783dede713bd9138e3495f9: Status 404 returned error can't find the container with id f6a5970cb6a8d63cbcf72b226015a24f279bda652783dede713bd9138e3495f9
Sep 16 18:39:21.684264 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:21.684238    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda206f1c7_dc3a_485e_8e7f_ddd6e09108f9.slice/crio-5863e17cb01d82bad4ca02f9cb95d25c7296c71cb830c15388ec0c75c9fbfa00 WatchSource:0}: Error finding container 5863e17cb01d82bad4ca02f9cb95d25c7296c71cb830c15388ec0c75c9fbfa00: Status 404 returned error can't find the container with id 5863e17cb01d82bad4ca02f9cb95d25c7296c71cb830c15388ec0c75c9fbfa00
Sep 16 18:39:21.686309 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:21.686291    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0930c4f1_458b_47e4_a122_d1080055cd14.slice/crio-b10e508a5269fb8d3264fd8bc18d41d48b012388b513d12d521ec56a96737364 WatchSource:0}: Error finding container b10e508a5269fb8d3264fd8bc18d41d48b012388b513d12d521ec56a96737364: Status 404 returned error can't find the container with id b10e508a5269fb8d3264fd8bc18d41d48b012388b513d12d521ec56a96737364
Sep 16 18:39:21.688987 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:21.687673    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b5df9d_b26c_4616_abc5_69b774ea6ea2.slice/crio-9b578d00149d37bf580d18180c387becfd70f598efb1715ce9e4e24ea972d8f8 WatchSource:0}: Error finding container 9b578d00149d37bf580d18180c387becfd70f598efb1715ce9e4e24ea972d8f8: Status 404 returned error can't find the container with id 9b578d00149d37bf580d18180c387becfd70f598efb1715ce9e4e24ea972d8f8
Sep 16 18:39:21.688987 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.688283    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:20 +0000 UTC" deadline="2028-03-24 14:33:53.863190299 +0000 UTC"
Sep 16 18:39:21.688987 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.688308    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13315h54m32.174886594s"
Sep 16 18:39:21.688987 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:21.688456    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa398820_2005_47bc_8687_b89d132bb6fb.slice/crio-014e64d4a6297d350f091f5be51db8e550bda5b8d56417fc5509489feb4de6ee WatchSource:0}: Error finding container 014e64d4a6297d350f091f5be51db8e550bda5b8d56417fc5509489feb4de6ee: Status 404 returned error can't find the container with id 014e64d4a6297d350f091f5be51db8e550bda5b8d56417fc5509489feb4de6ee
Sep 16 18:39:21.689500 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:21.689461    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960f29d3_0e56_4ca5_9f0a_d319b2107b04.slice/crio-69d3adc0e02e4656e37b24f79858b4a96f3625b9760b1ae77304faadae140998 WatchSource:0}: Error finding container 69d3adc0e02e4656e37b24f79858b4a96f3625b9760b1ae77304faadae140998: Status 404 returned error can't find the container with id 69d3adc0e02e4656e37b24f79858b4a96f3625b9760b1ae77304faadae140998
Sep 16 18:39:21.691666 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:21.691269    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c92186_ba53_4d0d_8c30_5dd1e3778b30.slice/crio-5f574d4ac2eb13a786df663a4a9086976ffa2c358be5ca3e7eb9dc452015c24b WatchSource:0}: Error finding container 5f574d4ac2eb13a786df663a4a9086976ffa2c358be5ca3e7eb9dc452015c24b: Status 404 returned error can't find the container with id 5f574d4ac2eb13a786df663a4a9086976ffa2c358be5ca3e7eb9dc452015c24b
Sep 16 18:39:21.692092 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:21.692069    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa975d83_17d8_4510_8ba1_0b0bf700f6bb.slice/crio-17adc421ae9241a7f6dee0a3123ede4dd8439debfe611743ee70efb4a7837ab9 WatchSource:0}: Error finding container 17adc421ae9241a7f6dee0a3123ede4dd8439debfe611743ee70efb4a7837ab9: Status 404 returned error can't find the container with id 17adc421ae9241a7f6dee0a3123ede4dd8439debfe611743ee70efb4a7837ab9
Sep 16 18:39:21.743428 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.743403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:21.743552 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:21.743495    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:21.753480 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.753446    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" event={"ID":"fa975d83-17d8-4510-8ba1-0b0bf700f6bb","Type":"ContainerStarted","Data":"17adc421ae9241a7f6dee0a3123ede4dd8439debfe611743ee70efb4a7837ab9"}
Sep 16 18:39:21.754735 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.754705    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-glc54" event={"ID":"960f29d3-0e56-4ca5-9f0a-d319b2107b04","Type":"ContainerStarted","Data":"69d3adc0e02e4656e37b24f79858b4a96f3625b9760b1ae77304faadae140998"}
Sep 16 18:39:21.756002 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.755974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pvshm" event={"ID":"fa398820-2005-47bc-8687-b89d132bb6fb","Type":"ContainerStarted","Data":"014e64d4a6297d350f091f5be51db8e550bda5b8d56417fc5509489feb4de6ee"}
Sep 16 18:39:21.757620 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.757595    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9wqg" event={"ID":"0930c4f1-458b-47e4-a122-d1080055cd14","Type":"ContainerStarted","Data":"b10e508a5269fb8d3264fd8bc18d41d48b012388b513d12d521ec56a96737364"}
Sep 16 18:39:21.758773 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.758756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dqm5k" event={"ID":"68b5df9d-b26c-4616-abc5-69b774ea6ea2","Type":"ContainerStarted","Data":"9b578d00149d37bf580d18180c387becfd70f598efb1715ce9e4e24ea972d8f8"}
Sep 16 18:39:21.760058 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.760026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-smzkz" event={"ID":"61c92186-ba53-4d0d-8c30-5dd1e3778b30","Type":"ContainerStarted","Data":"5f574d4ac2eb13a786df663a4a9086976ffa2c358be5ca3e7eb9dc452015c24b"}
Sep 16 18:39:21.761133 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.761114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx" event={"ID":"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9","Type":"ContainerStarted","Data":"5863e17cb01d82bad4ca02f9cb95d25c7296c71cb830c15388ec0c75c9fbfa00"}
Sep 16 18:39:21.762316 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.762291    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rrwk9" event={"ID":"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2","Type":"ContainerStarted","Data":"f6a5970cb6a8d63cbcf72b226015a24f279bda652783dede713bd9138e3495f9"}
Sep 16 18:39:21.763506 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:21.763488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khznh" event={"ID":"dda38eb1-7c5b-42cb-8557-640253d94a7f","Type":"ContainerStarted","Data":"a2fbd4bac525680718166221282454f7cdb6256c90467e3a894f0e6f2f3768db"}
Sep 16 18:39:22.262966 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.261986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:22.262966 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.262059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:22.262966 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.262252    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:22.262966 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.262319    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret podName:2447bda6-429c-48a3-ad0a-e8b58294e883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:24.262301023 +0000 UTC m=+6.050629382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret") pod "global-pull-secret-syncer-ztbzx" (UID: "2447bda6-429c-48a3-ad0a-e8b58294e883") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:22.262966 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.262788    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:22.262966 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.262881    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs podName:e08562f6-72fa-408c-9d9f-43ce0d4ced85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:24.262852406 +0000 UTC m=+6.051180773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs") pod "network-metrics-daemon-ds74d" (UID: "e08562f6-72fa-408c-9d9f-43ce0d4ced85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:22.363630 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.362952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:22.363630 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.363171    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:22.363630 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.363194    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:22.363630 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.363206    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kw8st for pod openshift-network-diagnostics/network-check-target-2xjm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:22.363630 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.363263    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st podName:43c61e8e-f211-4b63-afc1-99c5c0571682 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:24.363245789 +0000 UTC m=+6.151574148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kw8st" (UniqueName: "kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st") pod "network-check-target-2xjm7" (UID: "43c61e8e-f211-4b63-afc1-99c5c0571682") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:22.688588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.688428    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:20 +0000 UTC" deadline="2028-05-03 22:37:27.299055405 +0000 UTC"
Sep 16 18:39:22.688588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.688494    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14283h58m4.61056712s"
Sep 16 18:39:22.748026 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.747995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:22.748221 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.748193    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:22.748653 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.748633    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:22.748753 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:22.748731    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:22.790386 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.790346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal" event={"ID":"aa12150102fe3893c84359d7f70bc6ce","Type":"ContainerStarted","Data":"2238359affa3e96f6b5b8b4c2becf370ff61055f3ffd3696384833cf83f28371"}
Sep 16 18:39:22.795403 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.795362    2561 generic.go:356] "Generic (PLEG): container finished" podID="fb15a171a0a140f446df999ca2a1dbe0" containerID="28312c402ad3b66908faa4227002d77d7d9e626ca9c1c3723e24f3926c25c494" exitCode=0
Sep 16 18:39:22.795561 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.795423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal" event={"ID":"fb15a171a0a140f446df999ca2a1dbe0","Type":"ContainerDied","Data":"28312c402ad3b66908faa4227002d77d7d9e626ca9c1c3723e24f3926c25c494"}
Sep 16 18:39:22.812414 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:22.812353    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal" podStartSLOduration=2.812332625 podStartE2EDuration="2.812332625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:22.811613991 +0000 UTC m=+4.599942357" watchObservedRunningTime="2026-09-16 18:39:22.812332625 +0000 UTC m=+4.600660993"
Sep 16 18:39:23.744075 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:23.743992    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:23.744840 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:23.744132    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:23.821729 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:23.821645    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal" event={"ID":"fb15a171a0a140f446df999ca2a1dbe0","Type":"ContainerStarted","Data":"ea9a132aaf3835ed0fcc4f8173fd0008b4c5d00fd6bd3e476d346d8ea0351de1"}
Sep 16 18:39:24.282742 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:24.282687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:24.282934 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:24.282781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:24.283022 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.282939    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:24.283022 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.283004    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret podName:2447bda6-429c-48a3-ad0a-e8b58294e883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:28.282985494 +0000 UTC m=+10.071313860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret") pod "global-pull-secret-syncer-ztbzx" (UID: "2447bda6-429c-48a3-ad0a-e8b58294e883") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:24.283431 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.283410    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:24.283541 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.283471    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs podName:e08562f6-72fa-408c-9d9f-43ce0d4ced85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:28.283455069 +0000 UTC m=+10.071783430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs") pod "network-metrics-daemon-ds74d" (UID: "e08562f6-72fa-408c-9d9f-43ce0d4ced85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:24.383562 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:24.383512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:24.383757 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.383714    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:24.383757 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.383736    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:24.383757 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.383750    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kw8st for pod openshift-network-diagnostics/network-check-target-2xjm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:24.383906 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.383816    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st podName:43c61e8e-f211-4b63-afc1-99c5c0571682 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:28.383788834 +0000 UTC m=+10.172117182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kw8st" (UniqueName: "kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st") pod "network-check-target-2xjm7" (UID: "43c61e8e-f211-4b63-afc1-99c5c0571682") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:24.744745 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:24.744668    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:24.745208 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.744806    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:24.745208 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:24.744857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:24.745208 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:24.744985    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:25.743310 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:25.743272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:25.743491 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:25.743404    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:26.743409 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:26.743376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:26.743871 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:26.743524    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:26.743871 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:26.743368    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:26.743871 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:26.743820    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:27.743212 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:27.743178    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:27.743433 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:27.743309    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:28.322375 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:28.322338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:28.322595 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:28.322426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:28.322595 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.322495    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:28.322595 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.322524    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:28.322595 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.322584    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret podName:2447bda6-429c-48a3-ad0a-e8b58294e883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:36.322547757 +0000 UTC m=+18.110876117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret") pod "global-pull-secret-syncer-ztbzx" (UID: "2447bda6-429c-48a3-ad0a-e8b58294e883") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:28.322817 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.322604    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs podName:e08562f6-72fa-408c-9d9f-43ce0d4ced85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:36.322593788 +0000 UTC m=+18.110922138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs") pod "network-metrics-daemon-ds74d" (UID: "e08562f6-72fa-408c-9d9f-43ce0d4ced85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:28.422819 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:28.422779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:28.423003 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.422963    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:28.423003 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.422979    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:28.423003 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.422988    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kw8st for pod openshift-network-diagnostics/network-check-target-2xjm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:28.423179 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.423041    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st podName:43c61e8e-f211-4b63-afc1-99c5c0571682 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:36.423025363 +0000 UTC m=+18.211353710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kw8st" (UniqueName: "kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st") pod "network-check-target-2xjm7" (UID: "43c61e8e-f211-4b63-afc1-99c5c0571682") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:28.744801 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:28.744713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:28.745278 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.744834    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:28.745430 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:28.745409    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:28.745546 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:28.745525    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:29.744024 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:29.743989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:29.744199 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:29.744132    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:30.744159 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:30.744120    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:30.744655 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:30.744249    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:30.744655 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:30.744312    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:30.744655 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:30.744425    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:31.743327 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:31.743280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:31.743521 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:31.743403    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:32.744255 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:32.744217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:32.744731 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:32.744217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:32.744731 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:32.744355    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:32.744731 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:32.744414    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:33.743841 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:33.743620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:33.744014 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:33.743939    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:34.744041 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:34.744003    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:34.744468 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:34.744004    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:34.744468 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:34.744146    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:34.744468 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:34.744243    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:35.744015 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:35.743895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:35.744228 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:35.744035    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:36.380116 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:36.380083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:36.380322 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:36.380145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:36.380322 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.380250    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:36.380322 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.380257    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:36.380322 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.380310    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret podName:2447bda6-429c-48a3-ad0a-e8b58294e883 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:52.38029387 +0000 UTC m=+34.168622228 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret") pod "global-pull-secret-syncer-ztbzx" (UID: "2447bda6-429c-48a3-ad0a-e8b58294e883") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:36.380492 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.380329    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs podName:e08562f6-72fa-408c-9d9f-43ce0d4ced85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:52.380319691 +0000 UTC m=+34.168648035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs") pod "network-metrics-daemon-ds74d" (UID: "e08562f6-72fa-408c-9d9f-43ce0d4ced85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:36.481322 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:36.481291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:36.481499 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.481418    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:36.481499 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.481433    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:36.481499 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.481445    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kw8st for pod openshift-network-diagnostics/network-check-target-2xjm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:36.481660 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.481502    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st podName:43c61e8e-f211-4b63-afc1-99c5c0571682 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:52.481488222 +0000 UTC m=+34.269816566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kw8st" (UniqueName: "kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st") pod "network-check-target-2xjm7" (UID: "43c61e8e-f211-4b63-afc1-99c5c0571682") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:36.743725 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:36.743644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:36.743725 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:36.743680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:36.743923 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.743792    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:36.743979 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:36.743918    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:37.744046 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:37.744006    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:37.744512 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:37.744134    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:38.744585 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:38.744529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:38.745017 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:38.744660    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:38.745017 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:38.744716    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:38.745017 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:38.744826    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:39.743434 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.743366    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:39.743582 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:39.743459    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:39.857061 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.856825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pvshm" event={"ID":"fa398820-2005-47bc-8687-b89d132bb6fb","Type":"ContainerStarted","Data":"67fee743e1134444a776b498ad2d30c1a370289c11ae3aba3c522e922b789c3e"}
Sep 16 18:39:39.858292 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.858267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-smzkz" event={"ID":"61c92186-ba53-4d0d-8c30-5dd1e3778b30","Type":"ContainerStarted","Data":"d3b074275b0a90f4462ac3182c30a1660bd2d2aa91c78cc41a53cba677354530"}
Sep 16 18:39:39.859530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.859508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx" event={"ID":"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9","Type":"ContainerStarted","Data":"f006ad751f800572dc36b95b13076b88333a54d4e49229888f97788694977e1d"}
Sep 16 18:39:39.860900 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.860877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rrwk9" event={"ID":"46d448d3-6c08-4ec6-9d5d-c62c2635e0c2","Type":"ContainerStarted","Data":"056b86013ccbd149fd2c5e24a5868c3bf0d84bbca96084618aaff9a426d768d9"}
Sep 16 18:39:39.862719 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.862693    2561 generic.go:356] "Generic (PLEG): container finished" podID="dda38eb1-7c5b-42cb-8557-640253d94a7f" containerID="411d54963be66db3e12ea4548103dac96067a0f2c806f21611f5a43e9fe66733" exitCode=0
Sep 16 18:39:39.862837 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.862772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khznh" event={"ID":"dda38eb1-7c5b-42cb-8557-640253d94a7f","Type":"ContainerDied","Data":"411d54963be66db3e12ea4548103dac96067a0f2c806f21611f5a43e9fe66733"}
Sep 16 18:39:39.864992 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.864959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" event={"ID":"fa975d83-17d8-4510-8ba1-0b0bf700f6bb","Type":"ContainerStarted","Data":"bce111c48c7a4e1ed170cf1c532da2caf84287b2912b2bb9bd34737b4ad2e664"}
Sep 16 18:39:39.865080 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.865033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" event={"ID":"fa975d83-17d8-4510-8ba1-0b0bf700f6bb","Type":"ContainerStarted","Data":"a736e6cd77c388dc59f823d2c2e6a36814f32fb8a84d6f2ea4f138862962488a"}
Sep 16 18:39:39.867014 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.866989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-glc54" event={"ID":"960f29d3-0e56-4ca5-9f0a-d319b2107b04","Type":"ContainerStarted","Data":"0912771f311d2e59d81e550b016477578ec4ad14026e466310e559a0f815852c"}
Sep 16 18:39:39.885479 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.885428    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal" podStartSLOduration=19.885410956 podStartE2EDuration="19.885410956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:23.840844577 +0000 UTC m=+5.629172944" watchObservedRunningTime="2026-09-16 18:39:39.885410956 +0000 UTC m=+21.673739324"
Sep 16 18:39:39.900138 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.900097    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pvshm" podStartSLOduration=7.211425904 podStartE2EDuration="20.900080505s" totalImagesPullingTime="13.688654601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:39.884948636 +0000 UTC m=+21.673277005" watchObservedRunningTime="2026-09-16 18:39:39.900080505 +0000 UTC m=+21.688408870"
Sep 16 18:39:39.900311 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.900294    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rrwk9" podStartSLOduration=3.115016892 podStartE2EDuration="20.900288144s" totalImagesPullingTime="17.785271252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:39.89978131 +0000 UTC m=+21.688109677" watchObservedRunningTime="2026-09-16 18:39:39.900288144 +0000 UTC m=+21.688616509"
Sep 16 18:39:39.951168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.951102    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-smzkz" podStartSLOduration=3.176788468 podStartE2EDuration="20.951078549s" totalImagesPullingTime="17.774290081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:39.950642895 +0000 UTC m=+21.738971260" watchObservedRunningTime="2026-09-16 18:39:39.951078549 +0000 UTC m=+21.739406917"
Sep 16 18:39:39.982387 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:39.982344    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-glc54" podStartSLOduration=3.205905836 podStartE2EDuration="20.982325551s" totalImagesPullingTime="17.776419715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:39.981899515 +0000 UTC m=+21.770227891" watchObservedRunningTime="2026-09-16 18:39:39.982325551 +0000 UTC m=+21.770653916"
Sep 16 18:39:40.743796 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:40.743723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:40.743966 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:40.743726    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:40.743966 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:40.743871    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:40.743966 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:40.743941    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:40.874084 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:40.874042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" event={"ID":"fa975d83-17d8-4510-8ba1-0b0bf700f6bb","Type":"ContainerStarted","Data":"b4b6bb9e300ecba2ead367e12798dc061c81d95aae37d118753b646b91db2a35"}
Sep 16 18:39:40.874806 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:40.874142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" event={"ID":"fa975d83-17d8-4510-8ba1-0b0bf700f6bb","Type":"ContainerStarted","Data":"3f846e3841135b3155aa3a1d92cf230d5e9a66ad562d3725c59e86e90f922429"}
Sep 16 18:39:40.874806 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:40.874198    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" event={"ID":"fa975d83-17d8-4510-8ba1-0b0bf700f6bb","Type":"ContainerStarted","Data":"e62086320c50302ab03e9cab9896b85b81ce0413808872318bf75ca431206ffe"}
Sep 16 18:39:40.874806 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:40.874237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" event={"ID":"fa975d83-17d8-4510-8ba1-0b0bf700f6bb","Type":"ContainerStarted","Data":"2b7ef850f2386623e21dd31f890ba642c80dcfe7b51dc38d21f0dfc39ee81787"}
Sep 16 18:39:40.877700 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:40.877274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9wqg" event={"ID":"0930c4f1-458b-47e4-a122-d1080055cd14","Type":"ContainerStarted","Data":"28dcd68f9734d279a49dc194bd3313da8950fde7617f7ab2cbfcc7b35852d3cf"}
Sep 16 18:39:40.903685 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:40.903634    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n9wqg" podStartSLOduration=3.836804263 podStartE2EDuration="21.90361622s" totalImagesPullingTime="18.066811957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:40.903224346 +0000 UTC m=+22.691552714" watchObservedRunningTime="2026-09-16 18:39:40.90361622 +0000 UTC m=+22.691944586"
Sep 16 18:39:40.945611 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:40.945548    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:39:41.743905 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:41.743864    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:41.744185 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:41.744024    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:41.786559 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:41.786458    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:39:40.945604885Z","UUID":"4abb421f-e86a-4828-a8ce-708baa754f06","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:39:41.788171 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:41.788144    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:39:41.788296 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:41.788180    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:39:41.884817 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:41.884725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dqm5k" event={"ID":"68b5df9d-b26c-4616-abc5-69b774ea6ea2","Type":"ContainerStarted","Data":"ade37e1dc4ef82db35219cfd78528e93cf9216379ad26cb9df5e310b453d8482"}
Sep 16 18:39:41.887365 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:41.887334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx" event={"ID":"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9","Type":"ContainerStarted","Data":"993c90cc19d23f1f3c5a6b098cd69bc4317f890db10dd3b61de2cf3c46822594"}
Sep 16 18:39:42.743902 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:42.743648    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:42.744060 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:42.743684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:42.744060 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:42.743932    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:42.744178 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:42.744066    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:42.892110 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:42.892072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx" event={"ID":"a206f1c7-dc3a-485e-8e7f-ddd6e09108f9","Type":"ContainerStarted","Data":"ad1dbf60d6b6cba53270e1c897b843f6caec703fac69bf72ad521c1e77965c3e"}
Sep 16 18:39:42.896018 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:42.895988    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" event={"ID":"fa975d83-17d8-4510-8ba1-0b0bf700f6bb","Type":"ContainerStarted","Data":"6400f774e2abf506882670655ea61970c45e06eefcfb34cbca0c0ef41cdf7d9d"}
Sep 16 18:39:42.912644 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:42.912592    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dqm5k" podStartSLOduration=6.13458522 podStartE2EDuration="23.912575866s" totalImagesPullingTime="17.777990646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:41.901771553 +0000 UTC m=+23.690099923" watchObservedRunningTime="2026-09-16 18:39:42.912575866 +0000 UTC m=+24.700904223"
Sep 16 18:39:42.912870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:42.912848    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkpgx" podStartSLOduration=3.7519789279999998 podStartE2EDuration="23.912842242s" totalImagesPullingTime="20.160863314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:42.912439341 +0000 UTC m=+24.700767707" watchObservedRunningTime="2026-09-16 18:39:42.912842242 +0000 UTC m=+24.701170607"
Sep 16 18:39:43.376732 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:43.376696    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:43.377345 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:43.377321    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:43.743743 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:43.743666    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:43.743911 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:43.743771    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:44.743780 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:44.743526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:44.744231 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:44.743598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:44.744231 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:44.743849    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:44.744231 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:44.743954    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:44.903358 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:44.903320    2561 generic.go:356] "Generic (PLEG): container finished" podID="dda38eb1-7c5b-42cb-8557-640253d94a7f" containerID="eb9fa0b5b08076b84eb5103fe22299bc46b08cc26c4576be4443ef5f5a96af5b" exitCode=0
Sep 16 18:39:44.903358 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:44.903351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khznh" event={"ID":"dda38eb1-7c5b-42cb-8557-640253d94a7f","Type":"ContainerDied","Data":"eb9fa0b5b08076b84eb5103fe22299bc46b08cc26c4576be4443ef5f5a96af5b"}
Sep 16 18:39:44.907418 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:44.907391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" event={"ID":"fa975d83-17d8-4510-8ba1-0b0bf700f6bb","Type":"ContainerStarted","Data":"06061421c4465e0a9f02c5b86af38e80e69a7470589c1cbdcb0f33312f2915c5"}
Sep 16 18:39:44.907617 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:44.907595    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:44.922873 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:44.922852    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:44.965515 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:44.965462    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj" podStartSLOduration=8.168597711 podStartE2EDuration="25.965446063s" totalImagesPullingTime="17.796848352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:44.964191512 +0000 UTC m=+26.752519874" watchObservedRunningTime="2026-09-16 18:39:44.965446063 +0000 UTC m=+26.753774428"
Sep 16 18:39:45.744039 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:45.743998    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:45.744542 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:45.744134    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:45.909629 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:45.909601    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:45.909798 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:45.909678    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:45.924258 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:45.924232    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:39:46.743867 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.743832    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:46.743867 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.743867    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:46.744092 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:46.743968    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:46.744604 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:46.744083    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:46.891346 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.891313    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ds74d"]
Sep 16 18:39:46.894288 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.894259    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ztbzx"]
Sep 16 18:39:46.898349 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.898327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2xjm7"]
Sep 16 18:39:46.898497 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.898446    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:46.898553 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:46.898526    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:46.914406 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.914365    2561 generic.go:356] "Generic (PLEG): container finished" podID="dda38eb1-7c5b-42cb-8557-640253d94a7f" containerID="0b13f0abed2f67ebb8c7e647c9fa0dde0e250497c241fa81c20ffd16794a2ed6" exitCode=0
Sep 16 18:39:46.914539 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.914469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khznh" event={"ID":"dda38eb1-7c5b-42cb-8557-640253d94a7f","Type":"ContainerDied","Data":"0b13f0abed2f67ebb8c7e647c9fa0dde0e250497c241fa81c20ffd16794a2ed6"}
Sep 16 18:39:46.914539 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.914504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:46.914662 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:46.914560    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:46.914717 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:46.914688    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:46.914784 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:46.914756    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:48.744064 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:48.743986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:48.744492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:48.744069    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:48.744492 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:48.744088    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:48.744492 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:48.744178    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:48.744492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:48.744193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:48.744492 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:48.744281    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:48.759486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:48.759462    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:48.759613 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:48.759597    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:39:48.759982 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:48.759960    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pvshm"
Sep 16 18:39:48.921279 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:48.921246    2561 generic.go:356] "Generic (PLEG): container finished" podID="dda38eb1-7c5b-42cb-8557-640253d94a7f" containerID="32a0c34f4141744e6d7cd4f6a6b94170acc8e3f552da2e56b20dd18b4418c41d" exitCode=0
Sep 16 18:39:48.921424 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:48.921323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khznh" event={"ID":"dda38eb1-7c5b-42cb-8557-640253d94a7f","Type":"ContainerDied","Data":"32a0c34f4141744e6d7cd4f6a6b94170acc8e3f552da2e56b20dd18b4418c41d"}
Sep 16 18:39:50.743466 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:50.743275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:50.743946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:50.743275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:50.743946 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:50.743554    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xjm7" podUID="43c61e8e-f211-4b63-afc1-99c5c0571682"
Sep 16 18:39:50.743946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:50.743275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:50.743946 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:50.743690    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ds74d" podUID="e08562f6-72fa-408c-9d9f-43ce0d4ced85"
Sep 16 18:39:50.743946 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:50.743838    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztbzx" podUID="2447bda6-429c-48a3-ad0a-e8b58294e883"
Sep 16 18:39:52.400485 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.400448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:52.401013 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.400526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:52.401013 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.400626    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:52.401013 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.400699    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret podName:2447bda6-429c-48a3-ad0a-e8b58294e883 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:24.400681118 +0000 UTC m=+66.189009461 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret") pod "global-pull-secret-syncer-ztbzx" (UID: "2447bda6-429c-48a3-ad0a-e8b58294e883") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:52.401013 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.400635    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:52.401013 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.400781    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs podName:e08562f6-72fa-408c-9d9f-43ce0d4ced85 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:24.400766062 +0000 UTC m=+66.189094406 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs") pod "network-metrics-daemon-ds74d" (UID: "e08562f6-72fa-408c-9d9f-43ce0d4ced85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:52.501251 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.501155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:52.501387 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.501279    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:52.501387 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.501302    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:52.501387 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.501313    2561 projected.go:196] Error preparing data for projected volume kube-api-access-kw8st for pod openshift-network-diagnostics/network-check-target-2xjm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:52.501387 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.501370    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st podName:43c61e8e-f211-4b63-afc1-99c5c0571682 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:24.501354366 +0000 UTC m=+66.289682710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kw8st" (UniqueName: "kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st") pod "network-check-target-2xjm7" (UID: "43c61e8e-f211-4b63-afc1-99c5c0571682") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:52.545740 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.545715    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeReady"
Sep 16 18:39:52.545885 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.545857    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:39:52.604907 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.604875    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7df699999f-kr6dt"]
Sep 16 18:39:52.610750 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.610716    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.616839 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.616808    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:39:52.617109 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.616808    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:39:52.619794 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.618474    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:39:52.620357 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.620043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7smv5\""
Sep 16 18:39:52.628993 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.628961    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:39:52.652559 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.652519    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7df699999f-kr6dt"]
Sep 16 18:39:52.653132 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.653109    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gr4nw"]
Sep 16 18:39:52.657432 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.657409    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gr4nw"
Sep 16 18:39:52.659148 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.659115    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-glgft"]
Sep 16 18:39:52.660063 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.660034    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:39:52.660739 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.660716    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:39:52.660831 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.660755    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:39:52.661457 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.661437    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kf94w\""
Sep 16 18:39:52.663326 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.663309    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.667110 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.667090    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:39:52.667211 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.667143    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:39:52.667288 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.667263    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vxh4z\""
Sep 16 18:39:52.673922 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.673899    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gr4nw"]
Sep 16 18:39:52.689004 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.688980    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-glgft"]
Sep 16 18:39:52.702125 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.702098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtckk\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-kube-api-access-qtckk\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.702271 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.702138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ab8765f-7aac-443f-a9d5-c077f377f82c-ca-trust-extracted\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.702271 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.702169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-bound-sa-token\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.702271 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.702217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-installation-pull-secrets\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.702271 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.702261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-image-registry-private-configuration\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.702483 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.702292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-certificates\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.702483 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.702337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.702483 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.702366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-trusted-ca\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.744171 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.744137    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:39:52.744335 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.744137    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:39:52.744335 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.744151    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:39:52.747450 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.747355    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:39:52.747450 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.747387    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5zf4c\""
Sep 16 18:39:52.747649 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.747532    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcswh\""
Sep 16 18:39:52.747761 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.747746    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:39:52.747909 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.747888    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:39:52.748217 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.748196    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:39:52.802818 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.802776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-trusted-ca\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.802818 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.802823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a235aa-917b-41e2-b01a-6249d530b15e-tmp-dir\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.803042 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.802853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.803042 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.802879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb5hx\" (UniqueName: \"kubernetes.io/projected/15a235aa-917b-41e2-b01a-6249d530b15e-kube-api-access-pb5hx\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.803042 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.802907    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtckk\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-kube-api-access-qtckk\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.803042 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.802926    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ab8765f-7aac-443f-a9d5-c077f377f82c-ca-trust-extracted\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.803042 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.802953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-bound-sa-token\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.803042 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.802975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/15a235aa-917b-41e2-b01a-6249d530b15e-config-volume\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.803042 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.803000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-installation-pull-secrets\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.803042 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.803038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-image-registry-private-configuration\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.803431 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.803062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4mn\" (UniqueName: \"kubernetes.io/projected/bce3de21-b15d-4124-af2b-4457300f2fd1-kube-api-access-vw4mn\") pod \"ingress-canary-gr4nw\" (UID: \"bce3de21-b15d-4124-af2b-4457300f2fd1\") " pod="openshift-ingress-canary/ingress-canary-gr4nw"
Sep 16 18:39:52.803431 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.803093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-certificates\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.803431 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.803135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bce3de21-b15d-4124-af2b-4457300f2fd1-cert\") pod \"ingress-canary-gr4nw\" (UID: \"bce3de21-b15d-4124-af2b-4457300f2fd1\") " pod="openshift-ingress-canary/ingress-canary-gr4nw"
Sep 16 18:39:52.803431 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.803171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.803431 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.803258    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:39:52.803431 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.803272    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df699999f-kr6dt: secret "image-registry-tls" not found
Sep 16 18:39:52.803431 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.803325    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls podName:5ab8765f-7aac-443f-a9d5-c077f377f82c nodeName:}" failed. No retries permitted until 2026-09-16 18:39:53.30330545 +0000 UTC m=+35.091633797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls") pod "image-registry-7df699999f-kr6dt" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c") : secret "image-registry-tls" not found
Sep 16 18:39:52.803431 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.803363    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ab8765f-7aac-443f-a9d5-c077f377f82c-ca-trust-extracted\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.803705 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.803687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-certificates\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.804130 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.804112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-trusted-ca\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.807107 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.807087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-installation-pull-secrets\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.807178 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.807139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-image-registry-private-configuration\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.820685 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.820655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-bound-sa-token\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.821890 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.821872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtckk\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-kube-api-access-qtckk\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:52.903854 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.903810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/15a235aa-917b-41e2-b01a-6249d530b15e-config-volume\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.904035 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.903890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4mn\" (UniqueName: \"kubernetes.io/projected/bce3de21-b15d-4124-af2b-4457300f2fd1-kube-api-access-vw4mn\") pod \"ingress-canary-gr4nw\" (UID: \"bce3de21-b15d-4124-af2b-4457300f2fd1\") " pod="openshift-ingress-canary/ingress-canary-gr4nw"
Sep 16 18:39:52.904035 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.903956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bce3de21-b15d-4124-af2b-4457300f2fd1-cert\") pod \"ingress-canary-gr4nw\" (UID: \"bce3de21-b15d-4124-af2b-4457300f2fd1\") " pod="openshift-ingress-canary/ingress-canary-gr4nw"
Sep 16 18:39:52.904035 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.904013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a235aa-917b-41e2-b01a-6249d530b15e-tmp-dir\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.904179 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.904048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.904179 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.904083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb5hx\" (UniqueName: \"kubernetes.io/projected/15a235aa-917b-41e2-b01a-6249d530b15e-kube-api-access-pb5hx\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.904394 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.904367    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:39:52.904489 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:52.904438    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls podName:15a235aa-917b-41e2-b01a-6249d530b15e nodeName:}" failed. No retries permitted until 2026-09-16 18:39:53.404419956 +0000 UTC m=+35.192748300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls") pod "dns-default-glgft" (UID: "15a235aa-917b-41e2-b01a-6249d530b15e") : secret "dns-default-metrics-tls" not found
Sep 16 18:39:52.904489 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.904477    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a235aa-917b-41e2-b01a-6249d530b15e-tmp-dir\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.904617 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.904493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/15a235aa-917b-41e2-b01a-6249d530b15e-config-volume\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.906433 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.906409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bce3de21-b15d-4124-af2b-4457300f2fd1-cert\") pod \"ingress-canary-gr4nw\" (UID: \"bce3de21-b15d-4124-af2b-4457300f2fd1\") " pod="openshift-ingress-canary/ingress-canary-gr4nw"
Sep 16 18:39:52.918360 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.918329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb5hx\" (UniqueName: \"kubernetes.io/projected/15a235aa-917b-41e2-b01a-6249d530b15e-kube-api-access-pb5hx\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:52.918512 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.918494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4mn\" (UniqueName: \"kubernetes.io/projected/bce3de21-b15d-4124-af2b-4457300f2fd1-kube-api-access-vw4mn\") pod \"ingress-canary-gr4nw\" (UID: \"bce3de21-b15d-4124-af2b-4457300f2fd1\") " pod="openshift-ingress-canary/ingress-canary-gr4nw"
Sep 16 18:39:52.969556 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:52.969520    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gr4nw"
Sep 16 18:39:53.252607 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:53.252441    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gr4nw"]
Sep 16 18:39:53.255809 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:39:53.255777    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce3de21_b15d_4124_af2b_4457300f2fd1.slice/crio-ff5b21ea5a8db76c6784372e75c2fb5a2b7d43d0a6f296473f9f4052a25bfa34 WatchSource:0}: Error finding container ff5b21ea5a8db76c6784372e75c2fb5a2b7d43d0a6f296473f9f4052a25bfa34: Status 404 returned error can't find the container with id ff5b21ea5a8db76c6784372e75c2fb5a2b7d43d0a6f296473f9f4052a25bfa34
Sep 16 18:39:53.307563 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:53.307535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:53.307714 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:53.307693    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:39:53.307764 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:53.307716    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df699999f-kr6dt: secret "image-registry-tls" not found
Sep 16 18:39:53.307800 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:53.307773    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls podName:5ab8765f-7aac-443f-a9d5-c077f377f82c nodeName:}" failed. No retries permitted until 2026-09-16 18:39:54.30775739 +0000 UTC m=+36.096085739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls") pod "image-registry-7df699999f-kr6dt" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c") : secret "image-registry-tls" not found
Sep 16 18:39:53.408297 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:53.408260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:53.408779 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:53.408415    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:39:53.408779 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:53.408492    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls podName:15a235aa-917b-41e2-b01a-6249d530b15e nodeName:}" failed. No retries permitted until 2026-09-16 18:39:54.408477364 +0000 UTC m=+36.196805712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls") pod "dns-default-glgft" (UID: "15a235aa-917b-41e2-b01a-6249d530b15e") : secret "dns-default-metrics-tls" not found
Sep 16 18:39:53.937364 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:53.937332    2561 generic.go:356] "Generic (PLEG): container finished" podID="dda38eb1-7c5b-42cb-8557-640253d94a7f" containerID="5f2bd99b4ed5c6d16c955f3d6a5166bc9012ee2946bfe67a3900ccdc10e61be2" exitCode=0
Sep 16 18:39:53.937551 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:53.937419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khznh" event={"ID":"dda38eb1-7c5b-42cb-8557-640253d94a7f","Type":"ContainerDied","Data":"5f2bd99b4ed5c6d16c955f3d6a5166bc9012ee2946bfe67a3900ccdc10e61be2"}
Sep 16 18:39:53.938808 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:53.938781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gr4nw" event={"ID":"bce3de21-b15d-4124-af2b-4457300f2fd1","Type":"ContainerStarted","Data":"ff5b21ea5a8db76c6784372e75c2fb5a2b7d43d0a6f296473f9f4052a25bfa34"}
Sep 16 18:39:54.316395 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:54.316318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:54.316539 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:54.316452    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:39:54.316539 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:54.316467    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df699999f-kr6dt: secret "image-registry-tls" not found
Sep 16 18:39:54.316539 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:54.316532    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls podName:5ab8765f-7aac-443f-a9d5-c077f377f82c nodeName:}" failed. No retries permitted until 2026-09-16 18:39:56.316514425 +0000 UTC m=+38.104842782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls") pod "image-registry-7df699999f-kr6dt" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c") : secret "image-registry-tls" not found
Sep 16 18:39:54.417032 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:54.416996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:54.417627 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:54.417154    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:39:54.417627 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:54.417221    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls podName:15a235aa-917b-41e2-b01a-6249d530b15e nodeName:}" failed. No retries permitted until 2026-09-16 18:39:56.417203356 +0000 UTC m=+38.205531708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls") pod "dns-default-glgft" (UID: "15a235aa-917b-41e2-b01a-6249d530b15e") : secret "dns-default-metrics-tls" not found
Sep 16 18:39:54.945454 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:54.945409    2561 generic.go:356] "Generic (PLEG): container finished" podID="dda38eb1-7c5b-42cb-8557-640253d94a7f" containerID="f4fc1f5b8205cf3a7005f06b54b7a5efaef2b4d6bdbff701a6c97af77ff602f1" exitCode=0
Sep 16 18:39:54.945643 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:54.945480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khznh" event={"ID":"dda38eb1-7c5b-42cb-8557-640253d94a7f","Type":"ContainerDied","Data":"f4fc1f5b8205cf3a7005f06b54b7a5efaef2b4d6bdbff701a6c97af77ff602f1"}
Sep 16 18:39:55.951762 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:55.951723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khznh" event={"ID":"dda38eb1-7c5b-42cb-8557-640253d94a7f","Type":"ContainerStarted","Data":"438dc2942ca2988880aa6f4923ddeab2b15651cb23eaae0a5e562d33bbf70396"}
Sep 16 18:39:55.953293 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:55.953138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gr4nw" event={"ID":"bce3de21-b15d-4124-af2b-4457300f2fd1","Type":"ContainerStarted","Data":"8370676a07c5c68dc6da2c60c70211c7ca056af61015ef96a0af7c0fd41bba8d"}
Sep 16 18:39:55.987958 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:55.987903    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-khznh" podStartSLOduration=8.672817187 podStartE2EDuration="36.987882588s" totalImagesPullingTime="28.315065401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:55.981250028 +0000 UTC m=+37.769578393" watchObservedRunningTime="2026-09-16 18:39:55.987882588 +0000 UTC m=+37.776210953"
Sep 16 18:39:56.037927 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:56.037880    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gr4nw" podStartSLOduration=1.948112829 podStartE2EDuration="4.037863727s" totalImagesPullingTime="2.089750898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:56.03673601 +0000 UTC m=+37.825064375" watchObservedRunningTime="2026-09-16 18:39:56.037863727 +0000 UTC m=+37.826192130"
Sep 16 18:39:56.223582 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:56.223481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:56.333478 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:56.333437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:39:56.333651 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:56.333618    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:39:56.333651 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:56.333639    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df699999f-kr6dt: secret "image-registry-tls" not found
Sep 16 18:39:56.333728 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:56.333696    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls podName:5ab8765f-7aac-443f-a9d5-c077f377f82c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:00.333678885 +0000 UTC m=+42.122007229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls") pod "image-registry-7df699999f-kr6dt" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c") : secret "image-registry-tls" not found
Sep 16 18:39:56.434561 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:56.434525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:39:56.434700 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:56.434685    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:39:56.434756 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:39:56.434749    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls podName:15a235aa-917b-41e2-b01a-6249d530b15e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:00.434732002 +0000 UTC m=+42.223060347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls") pod "dns-default-glgft" (UID: "15a235aa-917b-41e2-b01a-6249d530b15e") : secret "dns-default-metrics-tls" not found
Sep 16 18:39:57.214613 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:57.214561    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:59.106466 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:59.106417    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:59.206527 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:59.206492    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:59.547087 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:59.547059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:59.607187 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:59.607152    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:59.668310 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:39:59.668277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:00.364091 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:00.364045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:40:00.364467 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:00.364208    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:00.364467 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:00.364224    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df699999f-kr6dt: secret "image-registry-tls" not found
Sep 16 18:40:00.364467 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:00.364292    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls podName:5ab8765f-7aac-443f-a9d5-c077f377f82c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:08.364274353 +0000 UTC m=+50.152602701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls") pod "image-registry-7df699999f-kr6dt" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c") : secret "image-registry-tls" not found
Sep 16 18:40:00.464606 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:00.464542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:40:00.464774 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:00.464709    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:00.464816 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:00.464775    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls podName:15a235aa-917b-41e2-b01a-6249d530b15e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:08.46475891 +0000 UTC m=+50.253087259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls") pod "dns-default-glgft" (UID: "15a235aa-917b-41e2-b01a-6249d530b15e") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:01.300199 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:01.300160    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:01.362027 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:01.361991    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:08.417792 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:08.417743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:40:08.418209 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:08.417904    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:08.418209 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:08.417920    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df699999f-kr6dt: secret "image-registry-tls" not found
Sep 16 18:40:08.418209 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:08.417974    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls podName:5ab8765f-7aac-443f-a9d5-c077f377f82c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:24.417957132 +0000 UTC m=+66.206285482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls") pod "image-registry-7df699999f-kr6dt" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c") : secret "image-registry-tls" not found
Sep 16 18:40:08.518890 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:08.518852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:40:08.519030 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:08.519003    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:08.519075 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:08.519065    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls podName:15a235aa-917b-41e2-b01a-6249d530b15e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:24.519051339 +0000 UTC m=+66.307379683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls") pod "dns-default-glgft" (UID: "15a235aa-917b-41e2-b01a-6249d530b15e") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:17.939922 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:17.939883    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x7hhj"
Sep 16 18:40:24.430156 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.430116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:40:24.430156 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.430158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:40:24.430633 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.430267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:40:24.430633 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:24.430386    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:24.430633 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:24.430407    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df699999f-kr6dt: secret "image-registry-tls" not found
Sep 16 18:40:24.430633 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:24.430469    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls podName:5ab8765f-7aac-443f-a9d5-c077f377f82c nodeName:}" failed. No retries permitted until 2026-09-16 18:40:56.430452837 +0000 UTC m=+98.218781197 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls") pod "image-registry-7df699999f-kr6dt" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c") : secret "image-registry-tls" not found
Sep 16 18:40:24.438324 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.438300    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:24.439476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.439457    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:24.441030 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:24.441015    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:40:24.441115 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:24.441068    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs podName:e08562f6-72fa-408c-9d9f-43ce0d4ced85 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:28.441054025 +0000 UTC m=+130.229382370 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs") pod "network-metrics-daemon-ds74d" (UID: "e08562f6-72fa-408c-9d9f-43ce0d4ced85") : secret "metrics-daemon-secret" not found
Sep 16 18:40:24.444089 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.444071    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2447bda6-429c-48a3-ad0a-e8b58294e883-original-pull-secret\") pod \"global-pull-secret-syncer-ztbzx\" (UID: \"2447bda6-429c-48a3-ad0a-e8b58294e883\") " pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:40:24.530613 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.530562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:40:24.530613 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.530613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:40:24.530854 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:24.530721    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:24.530854 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:24.530791    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls podName:15a235aa-917b-41e2-b01a-6249d530b15e nodeName:}" failed. No retries permitted until 2026-09-16 18:40:56.530775602 +0000 UTC m=+98.319103947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls") pod "dns-default-glgft" (UID: "15a235aa-917b-41e2-b01a-6249d530b15e") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:24.535089 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.535068    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:24.543451 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.543432    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:24.554025 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.553989    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw8st\" (UniqueName: \"kubernetes.io/projected/43c61e8e-f211-4b63-afc1-99c5c0571682-kube-api-access-kw8st\") pod \"network-check-target-2xjm7\" (UID: \"43c61e8e-f211-4b63-afc1-99c5c0571682\") " pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:40:24.555556 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.555541    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztbzx"
Sep 16 18:40:24.565319 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.565300    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5zf4c\""
Sep 16 18:40:24.573628 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.573612    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:40:24.851864 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.851830    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ztbzx"]
Sep 16 18:40:24.854864 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:40:24.854835    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2447bda6_429c_48a3_ad0a_e8b58294e883.slice/crio-ab10ef8fc1d7c8feda605fed74370b5cc06012e9212beb267dbc4df3800c9386 WatchSource:0}: Error finding container ab10ef8fc1d7c8feda605fed74370b5cc06012e9212beb267dbc4df3800c9386: Status 404 returned error can't find the container with id ab10ef8fc1d7c8feda605fed74370b5cc06012e9212beb267dbc4df3800c9386
Sep 16 18:40:24.875885 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:24.875857    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2xjm7"]
Sep 16 18:40:24.878667 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:40:24.878643    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c61e8e_f211_4b63_afc1_99c5c0571682.slice/crio-8f0210f06c3e167204d6b0342e5ed4e5f87fe99f8c7c20fe97fd0dfabefe528b WatchSource:0}: Error finding container 8f0210f06c3e167204d6b0342e5ed4e5f87fe99f8c7c20fe97fd0dfabefe528b: Status 404 returned error can't find the container with id 8f0210f06c3e167204d6b0342e5ed4e5f87fe99f8c7c20fe97fd0dfabefe528b
Sep 16 18:40:25.024544 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:25.024504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ztbzx" event={"ID":"2447bda6-429c-48a3-ad0a-e8b58294e883","Type":"ContainerStarted","Data":"ab10ef8fc1d7c8feda605fed74370b5cc06012e9212beb267dbc4df3800c9386"}
Sep 16 18:40:25.025617 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:25.025594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2xjm7" event={"ID":"43c61e8e-f211-4b63-afc1-99c5c0571682","Type":"ContainerStarted","Data":"8f0210f06c3e167204d6b0342e5ed4e5f87fe99f8c7c20fe97fd0dfabefe528b"}
Sep 16 18:40:28.122389 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.122351    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw"]
Sep 16 18:40:28.125337 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.125311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw"
Sep 16 18:40:28.131824 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.131771    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:28.131973 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.131930    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:28.132062 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.131964    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-w6286\""
Sep 16 18:40:28.148998 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.148955    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw"]
Sep 16 18:40:28.259612 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.259576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2z4w\" (UniqueName: \"kubernetes.io/projected/298c6c47-abf6-43c7-afb0-4068922e8803-kube-api-access-n2z4w\") pod \"volume-data-source-validator-86695b7cc7-pprnw\" (UID: \"298c6c47-abf6-43c7-afb0-4068922e8803\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw"
Sep 16 18:40:28.360611 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.360510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2z4w\" (UniqueName: \"kubernetes.io/projected/298c6c47-abf6-43c7-afb0-4068922e8803-kube-api-access-n2z4w\") pod \"volume-data-source-validator-86695b7cc7-pprnw\" (UID: \"298c6c47-abf6-43c7-afb0-4068922e8803\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw"
Sep 16 18:40:28.361547 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.361516    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-j9cpz"]
Sep 16 18:40:28.364389 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.364369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.369132 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.369104    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:40:28.369993 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.369972    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m7bzx\""
Sep 16 18:40:28.370933 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.370830    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:40:28.372025 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.371634    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:28.372709 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.372645    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:28.382135 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.382112    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:40:28.391896 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.391871    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-j9cpz"]
Sep 16 18:40:28.396028 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.396005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2z4w\" (UniqueName: \"kubernetes.io/projected/298c6c47-abf6-43c7-afb0-4068922e8803-kube-api-access-n2z4w\") pod \"volume-data-source-validator-86695b7cc7-pprnw\" (UID: \"298c6c47-abf6-43c7-afb0-4068922e8803\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw"
Sep 16 18:40:28.436552 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.436517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw"
Sep 16 18:40:28.562476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.562434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8sx\" (UniqueName: \"kubernetes.io/projected/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-kube-api-access-mt8sx\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.562644 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.562540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-serving-cert\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.562719 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.562642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-trusted-ca\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.562719 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.562699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-config\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.664095 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.664000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-serving-cert\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.664095 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.664085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-trusted-ca\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.664287 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.664133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-config\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.668546 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.664835    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8sx\" (UniqueName: \"kubernetes.io/projected/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-kube-api-access-mt8sx\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.668546 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.665198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-config\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.668546 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.665394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-trusted-ca\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.668546 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.667913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-serving-cert\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.682804 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.682777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8sx\" (UniqueName: \"kubernetes.io/projected/75109d51-2bb5-4be7-816d-3c4d5b5f9a5d-kube-api-access-mt8sx\") pod \"console-operator-696884fb9c-j9cpz\" (UID: \"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d\") " pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:28.975346 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:28.975310    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:29.728092 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:29.728049    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-j9cpz"]
Sep 16 18:40:29.729346 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:29.729327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw"]
Sep 16 18:40:29.731289 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:40:29.731256    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75109d51_2bb5_4be7_816d_3c4d5b5f9a5d.slice/crio-0f03a433daa44adfbd3f3d24f3a3d541e5b88c8747f94110db345552a8e4c153 WatchSource:0}: Error finding container 0f03a433daa44adfbd3f3d24f3a3d541e5b88c8747f94110db345552a8e4c153: Status 404 returned error can't find the container with id 0f03a433daa44adfbd3f3d24f3a3d541e5b88c8747f94110db345552a8e4c153
Sep 16 18:40:29.731973 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:40:29.731945    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298c6c47_abf6_43c7_afb0_4068922e8803.slice/crio-b0424b7dfdfcc4a8c944710f381f90753292279b67c2ece1d20ee2c3ac58759c WatchSource:0}: Error finding container b0424b7dfdfcc4a8c944710f381f90753292279b67c2ece1d20ee2c3ac58759c: Status 404 returned error can't find the container with id b0424b7dfdfcc4a8c944710f381f90753292279b67c2ece1d20ee2c3ac58759c
Sep 16 18:40:30.040765 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:30.040678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2xjm7" event={"ID":"43c61e8e-f211-4b63-afc1-99c5c0571682","Type":"ContainerStarted","Data":"be9eaf1e187823547bca6f57c219e957b802c71d55be778bb9c28e2e49922433"}
Sep 16 18:40:30.040908 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:30.040808    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:40:30.042117 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:30.042089    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ztbzx" event={"ID":"2447bda6-429c-48a3-ad0a-e8b58294e883","Type":"ContainerStarted","Data":"bf33d8cf9cafb3b6ffd984a55b6bf2b262905b17f15b3e4f126a51a50e78760c"}
Sep 16 18:40:30.043077 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:30.043051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" event={"ID":"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d","Type":"ContainerStarted","Data":"0f03a433daa44adfbd3f3d24f3a3d541e5b88c8747f94110db345552a8e4c153"}
Sep 16 18:40:30.043961 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:30.043943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw" event={"ID":"298c6c47-abf6-43c7-afb0-4068922e8803","Type":"ContainerStarted","Data":"b0424b7dfdfcc4a8c944710f381f90753292279b67c2ece1d20ee2c3ac58759c"}
Sep 16 18:40:30.065963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:30.065917    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2xjm7" podStartSLOduration=67.502946996 podStartE2EDuration="1m12.065899597s" totalImagesPullingTime="4.562952601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:30.06430252 +0000 UTC m=+71.852630885" watchObservedRunningTime="2026-09-16 18:40:30.065899597 +0000 UTC m=+71.854227962"
Sep 16 18:40:30.091308 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:30.091252    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ztbzx" podStartSLOduration=66.491892187 podStartE2EDuration="1m11.091232909s" totalImagesPullingTime="4.599340722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:30.090110138 +0000 UTC m=+71.878438505" watchObservedRunningTime="2026-09-16 18:40:30.091232909 +0000 UTC m=+71.879561277"
Sep 16 18:40:32.051301 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.051272    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/0.log"
Sep 16 18:40:32.051707 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.051312    2561 generic.go:356] "Generic (PLEG): container finished" podID="75109d51-2bb5-4be7-816d-3c4d5b5f9a5d" containerID="94f76de9c579157e43d067488ceb0d56b54c6b39b6c39e96ecddb3528839c365" exitCode=255
Sep 16 18:40:32.051707 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.051375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" event={"ID":"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d","Type":"ContainerDied","Data":"94f76de9c579157e43d067488ceb0d56b54c6b39b6c39e96ecddb3528839c365"}
Sep 16 18:40:32.051707 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.051702    2561 scope.go:117] "RemoveContainer" containerID="94f76de9c579157e43d067488ceb0d56b54c6b39b6c39e96ecddb3528839c365"
Sep 16 18:40:32.052992 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.052966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw" event={"ID":"298c6c47-abf6-43c7-afb0-4068922e8803","Type":"ContainerStarted","Data":"b51e912f6f838a3157f556af472108c7e2caf4355b2dd4ae2cffdb5fd52140be"}
Sep 16 18:40:32.105836 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.105788    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pprnw" podStartSLOduration=2.282626082 podStartE2EDuration="4.105770449s" totalImagesPullingTime="1.823144367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:32.10554325 +0000 UTC m=+73.893871638" watchObservedRunningTime="2026-09-16 18:40:32.105770449 +0000 UTC m=+73.894098805"
Sep 16 18:40:32.576170 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.576132    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hknqv"]
Sep 16 18:40:32.579071 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.579054    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.585635 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.585606    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:40:32.585635 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.585623    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:40:32.585801 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.585655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:40:32.585801 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.585624    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:40:32.586528 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.586511    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-txfhn\""
Sep 16 18:40:32.597018 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.596987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fa04cde-1113-4aaa-84af-1da4349e4b59-crio-socket\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.597139 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.597065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.597139 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.597093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4h24\" (UniqueName: \"kubernetes.io/projected/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-api-access-t4h24\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.597227 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.597149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fa04cde-1113-4aaa-84af-1da4349e4b59-data-volume\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.597227 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.597171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.598076 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.598053    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hknqv"]
Sep 16 18:40:32.698407 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.698363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fa04cde-1113-4aaa-84af-1da4349e4b59-crio-socket\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.698644 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.698450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.698644 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.698476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4h24\" (UniqueName: \"kubernetes.io/projected/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-api-access-t4h24\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.698644 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.698497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fa04cde-1113-4aaa-84af-1da4349e4b59-crio-socket\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.698644 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.698525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fa04cde-1113-4aaa-84af-1da4349e4b59-data-volume\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.698644 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.698549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.698644 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:32.698640    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:32.698883 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:32.698658    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:32.698883 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:32.698721    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.198699007 +0000 UTC m=+74.987027370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:32.698883 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:32.698751    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.198740074 +0000 UTC m=+74.987068433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:32.698988 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.698888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fa04cde-1113-4aaa-84af-1da4349e4b59-data-volume\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:32.712016 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:32.711985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4h24\" (UniqueName: \"kubernetes.io/projected/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-api-access-t4h24\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:33.058158 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:33.058128    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/1.log"
Sep 16 18:40:33.058599 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:33.058561    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/0.log"
Sep 16 18:40:33.058646 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:33.058610    2561 generic.go:356] "Generic (PLEG): container finished" podID="75109d51-2bb5-4be7-816d-3c4d5b5f9a5d" containerID="795283379bef66558ac54c1399e046dde72290b6f1eb66c75fd44461c354e743" exitCode=255
Sep 16 18:40:33.058680 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:33.058644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" event={"ID":"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d","Type":"ContainerDied","Data":"795283379bef66558ac54c1399e046dde72290b6f1eb66c75fd44461c354e743"}
Sep 16 18:40:33.058731 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:33.058717    2561 scope.go:117] "RemoveContainer" containerID="94f76de9c579157e43d067488ceb0d56b54c6b39b6c39e96ecddb3528839c365"
Sep 16 18:40:33.059009 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:33.058984    2561 scope.go:117] "RemoveContainer" containerID="795283379bef66558ac54c1399e046dde72290b6f1eb66c75fd44461c354e743"
Sep 16 18:40:33.059198 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:33.059171    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-j9cpz_openshift-console-operator(75109d51-2bb5-4be7-816d-3c4d5b5f9a5d)\"" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" podUID="75109d51-2bb5-4be7-816d-3c4d5b5f9a5d"
Sep 16 18:40:33.202083 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:33.202043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:33.202257 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:33.202123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:33.202257 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:33.202168    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:33.202257 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:33.202218    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:33.202257 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:33.202234    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.202218386 +0000 UTC m=+75.990546747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:33.202257 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:33.202256    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.202245593 +0000 UTC m=+75.990573936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:34.062427 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:34.062401    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/1.log"
Sep 16 18:40:34.062806 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:34.062736    2561 scope.go:117] "RemoveContainer" containerID="795283379bef66558ac54c1399e046dde72290b6f1eb66c75fd44461c354e743"
Sep 16 18:40:34.062911 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:34.062891    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-j9cpz_openshift-console-operator(75109d51-2bb5-4be7-816d-3c4d5b5f9a5d)\"" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" podUID="75109d51-2bb5-4be7-816d-3c4d5b5f9a5d"
Sep 16 18:40:34.208602 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:34.208548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:34.208760 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:34.208618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:34.208760 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:34.208681    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:34.208760 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:34.208687    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:34.208760 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:34.208730    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.208719047 +0000 UTC m=+77.997047405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:34.208760 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:34.208744    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.208737534 +0000 UTC m=+77.997065878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:34.494388 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:34.494359    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rrwk9_46d448d3-6c08-4ec6-9d5d-c62c2635e0c2/dns-node-resolver/0.log"
Sep 16 18:40:35.690480 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:35.690447    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-smzkz_61c92186-ba53-4d0d-8c30-5dd1e3778b30/node-ca/0.log"
Sep 16 18:40:36.222929 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:36.222877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:36.222929 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:36.222946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:36.223206 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:36.223027    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:36.223206 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:36.223052    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:36.223206 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:36.223081    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:40.223067229 +0000 UTC m=+82.011395577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:36.223206 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:36.223114    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:40.223097675 +0000 UTC m=+82.011426024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:36.292533 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:36.292500    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gr4nw_bce3de21-b15d-4124-af2b-4457300f2fd1/serve-healthcheck-canary/0.log"
Sep 16 18:40:38.975466 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:38.975432    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:38.975860 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:38.975486    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:38.975860 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:38.975809    2561 scope.go:117] "RemoveContainer" containerID="795283379bef66558ac54c1399e046dde72290b6f1eb66c75fd44461c354e743"
Sep 16 18:40:38.975964 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:38.975946    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-j9cpz_openshift-console-operator(75109d51-2bb5-4be7-816d-3c4d5b5f9a5d)\"" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" podUID="75109d51-2bb5-4be7-816d-3c4d5b5f9a5d"
Sep 16 18:40:40.253956 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:40.253906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:40.254335 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:40.253989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:40.254335 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:40.254066    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:40.254335 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:40.254130    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:48.254115536 +0000 UTC m=+90.042443883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:40.254335 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:40.254066    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:40.254335 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:40.254173    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:48.254163963 +0000 UTC m=+90.042492307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:41.044889 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.044779    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"]
Sep 16 18:40:41.048815 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.048799    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.052270 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.052246    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:40:41.052399 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.052313    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:40:41.052702 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.052686    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:40:41.053766 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.053750    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:40:41.075274 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.075244    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"]
Sep 16 18:40:41.160742 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.160700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5dqf\" (UniqueName: \"kubernetes.io/projected/063af880-981a-4711-a444-4a9525a8dfcf-kube-api-access-l5dqf\") pod \"klusterlet-addon-workmgr-67dffc9b6d-4m8z9\" (UID: \"063af880-981a-4711-a444-4a9525a8dfcf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.160957 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.160763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/063af880-981a-4711-a444-4a9525a8dfcf-klusterlet-config\") pod \"klusterlet-addon-workmgr-67dffc9b6d-4m8z9\" (UID: \"063af880-981a-4711-a444-4a9525a8dfcf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.160957 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.160913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063af880-981a-4711-a444-4a9525a8dfcf-tmp\") pod \"klusterlet-addon-workmgr-67dffc9b6d-4m8z9\" (UID: \"063af880-981a-4711-a444-4a9525a8dfcf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.189837 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.189799    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ssm7d"]
Sep 16 18:40:41.192869 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.192849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.197008 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.196981    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:40:41.197149 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.197066    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:40:41.197149 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.197094    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:40:41.199282 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.199256    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:40:41.205842 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.205816    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-d224v\""
Sep 16 18:40:41.209327 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.209305    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ssm7d"]
Sep 16 18:40:41.261354 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.261318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063af880-981a-4711-a444-4a9525a8dfcf-tmp\") pod \"klusterlet-addon-workmgr-67dffc9b6d-4m8z9\" (UID: \"063af880-981a-4711-a444-4a9525a8dfcf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.261735 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.261370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/676ce5f2-c307-450b-ab49-4ed908ff6024-signing-key\") pod \"service-ca-568869fdc5-ssm7d\" (UID: \"676ce5f2-c307-450b-ab49-4ed908ff6024\") " pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.261735 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.261412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8pdb\" (UniqueName: \"kubernetes.io/projected/676ce5f2-c307-450b-ab49-4ed908ff6024-kube-api-access-l8pdb\") pod \"service-ca-568869fdc5-ssm7d\" (UID: \"676ce5f2-c307-450b-ab49-4ed908ff6024\") " pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.261735 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.261435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/676ce5f2-c307-450b-ab49-4ed908ff6024-signing-cabundle\") pod \"service-ca-568869fdc5-ssm7d\" (UID: \"676ce5f2-c307-450b-ab49-4ed908ff6024\") " pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.261735 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.261453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5dqf\" (UniqueName: \"kubernetes.io/projected/063af880-981a-4711-a444-4a9525a8dfcf-kube-api-access-l5dqf\") pod \"klusterlet-addon-workmgr-67dffc9b6d-4m8z9\" (UID: \"063af880-981a-4711-a444-4a9525a8dfcf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.261735 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.261476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/063af880-981a-4711-a444-4a9525a8dfcf-klusterlet-config\") pod \"klusterlet-addon-workmgr-67dffc9b6d-4m8z9\" (UID: \"063af880-981a-4711-a444-4a9525a8dfcf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.261918 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.261743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/063af880-981a-4711-a444-4a9525a8dfcf-tmp\") pod \"klusterlet-addon-workmgr-67dffc9b6d-4m8z9\" (UID: \"063af880-981a-4711-a444-4a9525a8dfcf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.263893 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.263875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/063af880-981a-4711-a444-4a9525a8dfcf-klusterlet-config\") pod \"klusterlet-addon-workmgr-67dffc9b6d-4m8z9\" (UID: \"063af880-981a-4711-a444-4a9525a8dfcf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.273940 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.273911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5dqf\" (UniqueName: \"kubernetes.io/projected/063af880-981a-4711-a444-4a9525a8dfcf-kube-api-access-l5dqf\") pod \"klusterlet-addon-workmgr-67dffc9b6d-4m8z9\" (UID: \"063af880-981a-4711-a444-4a9525a8dfcf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.358199 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.358104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:41.362236 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.362218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/676ce5f2-c307-450b-ab49-4ed908ff6024-signing-cabundle\") pod \"service-ca-568869fdc5-ssm7d\" (UID: \"676ce5f2-c307-450b-ab49-4ed908ff6024\") " pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.362312 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.362295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/676ce5f2-c307-450b-ab49-4ed908ff6024-signing-key\") pod \"service-ca-568869fdc5-ssm7d\" (UID: \"676ce5f2-c307-450b-ab49-4ed908ff6024\") " pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.362370 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.362357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8pdb\" (UniqueName: \"kubernetes.io/projected/676ce5f2-c307-450b-ab49-4ed908ff6024-kube-api-access-l8pdb\") pod \"service-ca-568869fdc5-ssm7d\" (UID: \"676ce5f2-c307-450b-ab49-4ed908ff6024\") " pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.362876 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.362851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/676ce5f2-c307-450b-ab49-4ed908ff6024-signing-cabundle\") pod \"service-ca-568869fdc5-ssm7d\" (UID: \"676ce5f2-c307-450b-ab49-4ed908ff6024\") " pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.364786 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.364764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/676ce5f2-c307-450b-ab49-4ed908ff6024-signing-key\") pod \"service-ca-568869fdc5-ssm7d\" (UID: \"676ce5f2-c307-450b-ab49-4ed908ff6024\") " pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.377945 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.377919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8pdb\" (UniqueName: \"kubernetes.io/projected/676ce5f2-c307-450b-ab49-4ed908ff6024-kube-api-access-l8pdb\") pod \"service-ca-568869fdc5-ssm7d\" (UID: \"676ce5f2-c307-450b-ab49-4ed908ff6024\") " pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.501359 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.501305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-ssm7d"
Sep 16 18:40:41.635624 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.635435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"]
Sep 16 18:40:41.637984 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:40:41.637952    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063af880_981a_4711_a444_4a9525a8dfcf.slice/crio-94ca31cc736a94ee39ef6291cb373b520a1e2a7c740c77abe9a80ade6aadd330 WatchSource:0}: Error finding container 94ca31cc736a94ee39ef6291cb373b520a1e2a7c740c77abe9a80ade6aadd330: Status 404 returned error can't find the container with id 94ca31cc736a94ee39ef6291cb373b520a1e2a7c740c77abe9a80ade6aadd330
Sep 16 18:40:41.778986 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:41.778943    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ssm7d"]
Sep 16 18:40:41.781350 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:40:41.781321    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676ce5f2_c307_450b_ab49_4ed908ff6024.slice/crio-f4b8f35405436d384f0abd18a16506fd371a97233e4af39e01b9c89af2af23a5 WatchSource:0}: Error finding container f4b8f35405436d384f0abd18a16506fd371a97233e4af39e01b9c89af2af23a5: Status 404 returned error can't find the container with id f4b8f35405436d384f0abd18a16506fd371a97233e4af39e01b9c89af2af23a5
Sep 16 18:40:42.083213 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:42.083176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-ssm7d" event={"ID":"676ce5f2-c307-450b-ab49-4ed908ff6024","Type":"ContainerStarted","Data":"f4b8f35405436d384f0abd18a16506fd371a97233e4af39e01b9c89af2af23a5"}
Sep 16 18:40:42.084330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:42.084296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9" event={"ID":"063af880-981a-4711-a444-4a9525a8dfcf","Type":"ContainerStarted","Data":"94ca31cc736a94ee39ef6291cb373b520a1e2a7c740c77abe9a80ade6aadd330"}
Sep 16 18:40:46.099235 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:46.099194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-ssm7d" event={"ID":"676ce5f2-c307-450b-ab49-4ed908ff6024","Type":"ContainerStarted","Data":"cd68d062c9cff6dd7f5e1b8513d1f1da2e2a293d18027bf358ed18f9c16b3128"}
Sep 16 18:40:46.100862 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:46.100837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9" event={"ID":"063af880-981a-4711-a444-4a9525a8dfcf","Type":"ContainerStarted","Data":"530ba6d395e286f0d7ac64c11b36ed129886751cf1666d354722a2e5453364d5"}
Sep 16 18:40:46.101054 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:46.101039    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:46.102529 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:46.102510    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9"
Sep 16 18:40:46.128816 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:46.128768    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-ssm7d" podStartSLOduration=2.542265941 podStartE2EDuration="5.128751691s" totalImagesPullingTime="2.58648575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:46.127760445 +0000 UTC m=+87.916088812" watchObservedRunningTime="2026-09-16 18:40:46.128751691 +0000 UTC m=+87.917080056"
Sep 16 18:40:48.325878 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:48.325837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:48.325878 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:48.325887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:48.326415 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:48.325998    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:48.326415 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:48.326084    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls podName:9fa04cde-1113-4aaa-84af-1da4349e4b59 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:04.326063165 +0000 UTC m=+106.114391509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hknqv" (UID: "9fa04cde-1113-4aaa-84af-1da4349e4b59") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:48.326415 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:48.326402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fa04cde-1113-4aaa-84af-1da4349e4b59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:40:53.743648 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:53.743611    2561 scope.go:117] "RemoveContainer" containerID="795283379bef66558ac54c1399e046dde72290b6f1eb66c75fd44461c354e743"
Sep 16 18:40:54.129512 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:54.129435    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:40:54.129928 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:54.129909    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/1.log"
Sep 16 18:40:54.129990 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:54.129951    2561 generic.go:356] "Generic (PLEG): container finished" podID="75109d51-2bb5-4be7-816d-3c4d5b5f9a5d" containerID="892ddfac8d704d9c3506030db54fb40123e5ad7c631c60f5edb730fb0ffc5dab" exitCode=255
Sep 16 18:40:54.130026 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:54.129989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" event={"ID":"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d","Type":"ContainerDied","Data":"892ddfac8d704d9c3506030db54fb40123e5ad7c631c60f5edb730fb0ffc5dab"}
Sep 16 18:40:54.130059 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:54.130044    2561 scope.go:117] "RemoveContainer" containerID="795283379bef66558ac54c1399e046dde72290b6f1eb66c75fd44461c354e743"
Sep 16 18:40:54.130420 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:54.130401    2561 scope.go:117] "RemoveContainer" containerID="892ddfac8d704d9c3506030db54fb40123e5ad7c631c60f5edb730fb0ffc5dab"
Sep 16 18:40:54.130596 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:54.130562    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-j9cpz_openshift-console-operator(75109d51-2bb5-4be7-816d-3c4d5b5f9a5d)\"" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" podUID="75109d51-2bb5-4be7-816d-3c4d5b5f9a5d"
Sep 16 18:40:54.149923 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:54.149874    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dffc9b6d-4m8z9" podStartSLOduration=9.22153735 podStartE2EDuration="13.149855867s" totalImagesPullingTime="3.928318517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:46.15419655 +0000 UTC m=+87.942524917" watchObservedRunningTime="2026-09-16 18:40:54.149855867 +0000 UTC m=+95.938184233"
Sep 16 18:40:55.134246 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:55.134217    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:40:56.489828 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:56.489802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:40:56.492241 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:56.492217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"image-registry-7df699999f-kr6dt\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") " pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:40:56.528047 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:56.528020    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7smv5\""
Sep 16 18:40:56.534071 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:56.534049    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:40:56.591656 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:56.590935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:40:56.601163 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:56.601110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/15a235aa-917b-41e2-b01a-6249d530b15e-metrics-tls\") pod \"dns-default-glgft\" (UID: \"15a235aa-917b-41e2-b01a-6249d530b15e\") " pod="openshift-dns/dns-default-glgft"
Sep 16 18:40:56.838320 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:56.838290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7df699999f-kr6dt"]
Sep 16 18:40:56.839873 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:40:56.839845    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab8765f_7aac_443f_a9d5_c077f377f82c.slice/crio-239ed28d34eb70d13b9b2fbf47385155db91aebdab65cb47079da049563b3632 WatchSource:0}: Error finding container 239ed28d34eb70d13b9b2fbf47385155db91aebdab65cb47079da049563b3632: Status 404 returned error can't find the container with id 239ed28d34eb70d13b9b2fbf47385155db91aebdab65cb47079da049563b3632
Sep 16 18:40:56.879199 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:56.879178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vxh4z\""
Sep 16 18:40:56.887537 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:56.887514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glgft"
Sep 16 18:40:57.142802 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:57.142723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df699999f-kr6dt" event={"ID":"5ab8765f-7aac-443f-a9d5-c077f377f82c","Type":"ContainerStarted","Data":"188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d"}
Sep 16 18:40:57.142802 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:57.142770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df699999f-kr6dt" event={"ID":"5ab8765f-7aac-443f-a9d5-c077f377f82c","Type":"ContainerStarted","Data":"239ed28d34eb70d13b9b2fbf47385155db91aebdab65cb47079da049563b3632"}
Sep 16 18:40:57.142802 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:57.142801    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:40:57.185703 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:57.185668    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-glgft"]
Sep 16 18:40:57.189324 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:40:57.189292    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a235aa_917b_41e2_b01a_6249d530b15e.slice/crio-ec10a981bd87337c1d22ca295f67b357d14f24fcb263aedaa8d77682c297705e WatchSource:0}: Error finding container ec10a981bd87337c1d22ca295f67b357d14f24fcb263aedaa8d77682c297705e: Status 404 returned error can't find the container with id ec10a981bd87337c1d22ca295f67b357d14f24fcb263aedaa8d77682c297705e
Sep 16 18:40:57.195118 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:57.195022    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7df699999f-kr6dt" podStartSLOduration=98.195002344 podStartE2EDuration="1m38.195002344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:57.194081686 +0000 UTC m=+98.982410052" watchObservedRunningTime="2026-09-16 18:40:57.195002344 +0000 UTC m=+98.983330710"
Sep 16 18:40:58.147811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:58.147764    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glgft" event={"ID":"15a235aa-917b-41e2-b01a-6249d530b15e","Type":"ContainerStarted","Data":"ec10a981bd87337c1d22ca295f67b357d14f24fcb263aedaa8d77682c297705e"}
Sep 16 18:40:58.975882 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:58.975842    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:58.976053 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:58.975905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:40:58.976248 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:40:58.976233    2561 scope.go:117] "RemoveContainer" containerID="892ddfac8d704d9c3506030db54fb40123e5ad7c631c60f5edb730fb0ffc5dab"
Sep 16 18:40:58.976432 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:40:58.976411    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-j9cpz_openshift-console-operator(75109d51-2bb5-4be7-816d-3c4d5b5f9a5d)\"" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" podUID="75109d51-2bb5-4be7-816d-3c4d5b5f9a5d"
Sep 16 18:41:00.155406 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:00.155373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glgft" event={"ID":"15a235aa-917b-41e2-b01a-6249d530b15e","Type":"ContainerStarted","Data":"eb6fdf03b323848469066b190e8c8752708f04e320e09bcf9e57051e1317a524"}
Sep 16 18:41:00.155789 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:00.155427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glgft" event={"ID":"15a235aa-917b-41e2-b01a-6249d530b15e","Type":"ContainerStarted","Data":"1efe99540b10da5f86b1f47b55d1d8f7ad635bf4139a22d1efb47fd05511605e"}
Sep 16 18:41:00.155789 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:00.155463    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-glgft"
Sep 16 18:41:00.178607 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:00.178532    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-glgft" podStartSLOduration=66.331615259 podStartE2EDuration="1m8.178514667s" totalImagesPullingTime="1.846899408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:00.177064953 +0000 UTC m=+101.965393354" watchObservedRunningTime="2026-09-16 18:41:00.178514667 +0000 UTC m=+101.966843033"
Sep 16 18:41:01.049671 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:01.049643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2xjm7"
Sep 16 18:41:04.349347 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:04.349302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:41:04.351682 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:04.351661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fa04cde-1113-4aaa-84af-1da4349e4b59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hknqv\" (UID: \"9fa04cde-1113-4aaa-84af-1da4349e4b59\") " pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:41:04.387720 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:04.387671    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hknqv"
Sep 16 18:41:04.673133 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:04.673093    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa04cde_1113_4aaa_84af_1da4349e4b59.slice/crio-f65915e444253977263c2ad65a3c1b03375b10d368e9b529fb8e1237ec4f1703 WatchSource:0}: Error finding container f65915e444253977263c2ad65a3c1b03375b10d368e9b529fb8e1237ec4f1703: Status 404 returned error can't find the container with id f65915e444253977263c2ad65a3c1b03375b10d368e9b529fb8e1237ec4f1703
Sep 16 18:41:04.673253 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:04.673213    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hknqv"]
Sep 16 18:41:05.173438 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:05.173406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hknqv" event={"ID":"9fa04cde-1113-4aaa-84af-1da4349e4b59","Type":"ContainerStarted","Data":"3acd7ef6d8e9f1b218601fb0d14246d18f25b654fe16be8ae61d6a673d6467dc"}
Sep 16 18:41:05.173631 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:05.173461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hknqv" event={"ID":"9fa04cde-1113-4aaa-84af-1da4349e4b59","Type":"ContainerStarted","Data":"f65915e444253977263c2ad65a3c1b03375b10d368e9b529fb8e1237ec4f1703"}
Sep 16 18:41:06.179946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:06.179905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hknqv" event={"ID":"9fa04cde-1113-4aaa-84af-1da4349e4b59","Type":"ContainerStarted","Data":"9c4f3a5aa1e85527800b596436dc1c0cea9c55c8a62344008f9f1db7aea7163c"}
Sep 16 18:41:07.185386 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:07.185349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hknqv" event={"ID":"9fa04cde-1113-4aaa-84af-1da4349e4b59","Type":"ContainerStarted","Data":"c35f09fd73ecfb7ff6b94c9925fa175b961e71953e47ece38b3be75bef1a3e73"}
Sep 16 18:41:09.744355 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.744317    2561 scope.go:117] "RemoveContainer" containerID="892ddfac8d704d9c3506030db54fb40123e5ad7c631c60f5edb730fb0ffc5dab"
Sep 16 18:41:09.744773 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:09.744525    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-j9cpz_openshift-console-operator(75109d51-2bb5-4be7-816d-3c4d5b5f9a5d)\"" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" podUID="75109d51-2bb5-4be7-816d-3c4d5b5f9a5d"
Sep 16 18:41:09.921255 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.921195    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hknqv" podStartSLOduration=35.774953411 podStartE2EDuration="37.921173456s" totalImagesPullingTime="2.146220045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:07.253743874 +0000 UTC m=+109.042072239" watchObservedRunningTime="2026-09-16 18:41:09.921173456 +0000 UTC m=+111.709501823"
Sep 16 18:41:09.921599 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.921582    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"]
Sep 16 18:41:09.924942 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.924925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:09.941127 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.941106    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:41:09.941899 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.941881    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:41:09.947214 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.947197    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:41:09.947290 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.947204    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:41:09.955032 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.955014    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:41:09.955426 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.955410    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xkm48\""
Sep 16 18:41:09.973792 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.973760    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"]
Sep 16 18:41:09.988040 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.988004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cd53286-79c3-4788-b928-e822171d87ab-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:09.988040 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.988041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mn2g\" (UniqueName: \"kubernetes.io/projected/1cd53286-79c3-4788-b928-e822171d87ab-kube-api-access-4mn2g\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:09.988280 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.988081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd53286-79c3-4788-b928-e822171d87ab-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:09.988280 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:09.988194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cd53286-79c3-4788-b928-e822171d87ab-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.089326 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.089225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cd53286-79c3-4788-b928-e822171d87ab-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.089326 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.089269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cd53286-79c3-4788-b928-e822171d87ab-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.089326 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.089290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mn2g\" (UniqueName: \"kubernetes.io/projected/1cd53286-79c3-4788-b928-e822171d87ab-kube-api-access-4mn2g\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.089659 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.089441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd53286-79c3-4788-b928-e822171d87ab-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.090090 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.090065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd53286-79c3-4788-b928-e822171d87ab-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.091758 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.091729    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cd53286-79c3-4788-b928-e822171d87ab-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.091849 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.091732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cd53286-79c3-4788-b928-e822171d87ab-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.099825 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.099804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mn2g\" (UniqueName: \"kubernetes.io/projected/1cd53286-79c3-4788-b928-e822171d87ab-kube-api-access-4mn2g\") pod \"prometheus-operator-68bcd7f987-nl5x9\" (UID: \"1cd53286-79c3-4788-b928-e822171d87ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.161275 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.161245    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-glgft"
Sep 16 18:41:10.233586 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.233540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"
Sep 16 18:41:10.524233 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:10.524210    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9"]
Sep 16 18:41:10.526184 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:10.526156    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd53286_79c3_4788_b928_e822171d87ab.slice/crio-8bb64bb66d9d4427e66fdd522f6bad597e208173e29952a344510ac6f83455d2 WatchSource:0}: Error finding container 8bb64bb66d9d4427e66fdd522f6bad597e208173e29952a344510ac6f83455d2: Status 404 returned error can't find the container with id 8bb64bb66d9d4427e66fdd522f6bad597e208173e29952a344510ac6f83455d2
Sep 16 18:41:11.200384 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:11.200346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9" event={"ID":"1cd53286-79c3-4788-b928-e822171d87ab","Type":"ContainerStarted","Data":"8bb64bb66d9d4427e66fdd522f6bad597e208173e29952a344510ac6f83455d2"}
Sep 16 18:41:12.209729 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:12.209647    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9" event={"ID":"1cd53286-79c3-4788-b928-e822171d87ab","Type":"ContainerStarted","Data":"dc5eb589607532fe60e22325f704abc9f59412a37970efb70f8c7c354e490436"}
Sep 16 18:41:12.210079 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:12.209730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9" event={"ID":"1cd53286-79c3-4788-b928-e822171d87ab","Type":"ContainerStarted","Data":"c5b52ec5ff9ead713350cb100b603772eeb1b86cf462c8cd87c4fb64b369d357"}
Sep 16 18:41:12.238796 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:12.238741    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-nl5x9" podStartSLOduration=2.009687063 podStartE2EDuration="3.238724678s" totalImagesPullingTime="1.229037615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:12.237812246 +0000 UTC m=+114.026140612" watchObservedRunningTime="2026-09-16 18:41:12.238724678 +0000 UTC m=+114.027053043"
Sep 16 18:41:14.885711 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.885674    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h95tc"]
Sep 16 18:41:14.893231 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.893210    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:14.897784 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.897760    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:41:14.898028 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.898013    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:41:14.898183 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.898164    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cxq8v\""
Sep 16 18:41:14.898648 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.898630    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:41:14.964762 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.964731    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"]
Sep 16 18:41:14.968421 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.968404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:14.971093 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.971065    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:41:14.971210 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.971190    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jc6kr\""
Sep 16 18:41:14.971587 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.971551    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:41:14.971681 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:14.971590    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:41:15.010200 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.010170    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"]
Sep 16 18:41:15.030168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.030132    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4701e0a7-c118-47b8-9ff7-d38c38a7794d-sys\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.030168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.030167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-wtmp\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.030391 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.030191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.030391 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.030266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-tls\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.030391 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.030308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-accelerators-collector-config\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.030391 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.030339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-textfile\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.030524 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.030438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svng6\" (UniqueName: \"kubernetes.io/projected/4701e0a7-c118-47b8-9ff7-d38c38a7794d-kube-api-access-svng6\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.030524 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.030465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4701e0a7-c118-47b8-9ff7-d38c38a7794d-root\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.030524 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.030481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4701e0a7-c118-47b8-9ff7-d38c38a7794d-metrics-client-ca\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131062 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svng6\" (UniqueName: \"kubernetes.io/projected/4701e0a7-c118-47b8-9ff7-d38c38a7794d-kube-api-access-svng6\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131062 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.131330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4701e0a7-c118-47b8-9ff7-d38c38a7794d-root\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4701e0a7-c118-47b8-9ff7-d38c38a7794d-metrics-client-ca\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131151    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4701e0a7-c118-47b8-9ff7-d38c38a7794d-sys\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-wtmp\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4701e0a7-c118-47b8-9ff7-d38c38a7794d-root\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsttb\" (UniqueName: \"kubernetes.io/projected/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-api-access-qsttb\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.131330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4701e0a7-c118-47b8-9ff7-d38c38a7794d-sys\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.131811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-wtmp\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-tls\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-accelerators-collector-config\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a1a36bd-4778-4ef6-8bc8-6894080ed305-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.131811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-textfile\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.131811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.131811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a1a36bd-4778-4ef6-8bc8-6894080ed305-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.131811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-textfile\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.132204 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4701e0a7-c118-47b8-9ff7-d38c38a7794d-metrics-client-ca\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.132204 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.131925    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-accelerators-collector-config\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.133604 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.133584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.133883 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.133860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4701e0a7-c118-47b8-9ff7-d38c38a7794d-node-exporter-tls\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.155510 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.155456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svng6\" (UniqueName: \"kubernetes.io/projected/4701e0a7-c118-47b8-9ff7-d38c38a7794d-kube-api-access-svng6\") pod \"node-exporter-h95tc\" (UID: \"4701e0a7-c118-47b8-9ff7-d38c38a7794d\") " pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.202622 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.202587    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h95tc"
Sep 16 18:41:15.216290 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:15.216261    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4701e0a7_c118_47b8_9ff7_d38c38a7794d.slice/crio-2d44dde760b1a91af3331db6e85accb397655a8149b20c0f4b005d5dd0bb0f5f WatchSource:0}: Error finding container 2d44dde760b1a91af3331db6e85accb397655a8149b20c0f4b005d5dd0bb0f5f: Status 404 returned error can't find the container with id 2d44dde760b1a91af3331db6e85accb397655a8149b20c0f4b005d5dd0bb0f5f
Sep 16 18:41:15.221589 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.221544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h95tc" event={"ID":"4701e0a7-c118-47b8-9ff7-d38c38a7794d","Type":"ContainerStarted","Data":"2d44dde760b1a91af3331db6e85accb397655a8149b20c0f4b005d5dd0bb0f5f"}
Sep 16 18:41:15.233003 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.232978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.233130 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.233018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsttb\" (UniqueName: \"kubernetes.io/projected/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-api-access-qsttb\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.233130 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.233042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.233130 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.233075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a1a36bd-4778-4ef6-8bc8-6894080ed305-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.233130 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.233097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.233130 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.233120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a1a36bd-4778-4ef6-8bc8-6894080ed305-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.233530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.233492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a1a36bd-4778-4ef6-8bc8-6894080ed305-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.233875 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.233853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a1a36bd-4778-4ef6-8bc8-6894080ed305-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.233875 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.233862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.235421 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.235395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.235497 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.235477    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.273678 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.273650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsttb\" (UniqueName: \"kubernetes.io/projected/5a1a36bd-4778-4ef6-8bc8-6894080ed305-kube-api-access-qsttb\") pod \"kube-state-metrics-59fbdd77d-hc2fz\" (UID: \"5a1a36bd-4778-4ef6-8bc8-6894080ed305\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.277453 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.277432    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"
Sep 16 18:41:15.566963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:15.566932    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz"]
Sep 16 18:41:15.568754 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:15.568721    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1a36bd_4778_4ef6_8bc8_6894080ed305.slice/crio-d05517c5750751690bae1285c35d352c15df23bb82350c6b5fc59501e6f7ff67 WatchSource:0}: Error finding container d05517c5750751690bae1285c35d352c15df23bb82350c6b5fc59501e6f7ff67: Status 404 returned error can't find the container with id d05517c5750751690bae1285c35d352c15df23bb82350c6b5fc59501e6f7ff67
Sep 16 18:41:16.169368 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.169336    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:41:16.172716 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.172695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.177601 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.177562    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:41:16.177855 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.177827    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:41:16.177972 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.177864    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:41:16.177972 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.177945    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qljqg\""
Sep 16 18:41:16.180421 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.180403    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:41:16.180526 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.180469    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:41:16.180616 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.180597    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:41:16.188470 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.188455    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:41:16.188535 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.188502    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:41:16.217869 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.217846    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:41:16.226465 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.226438    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz" event={"ID":"5a1a36bd-4778-4ef6-8bc8-6894080ed305","Type":"ContainerStarted","Data":"d05517c5750751690bae1285c35d352c15df23bb82350c6b5fc59501e6f7ff67"}
Sep 16 18:41:16.238984 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.238954    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:41:16.242596 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242549    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.242712 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.242712 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242676    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.242830 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242741    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.242830 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.242830 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mj5\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-kube-api-access-g4mj5\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.242976 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-config-out\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.242976 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242906    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.242976 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.242976 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.242964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.243168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.243007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-config-volume\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.243168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.243021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.243168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.243042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-web-config\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344520 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344473    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-config-volume\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344520 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-web-config\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344739    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mj5\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-kube-api-access-g4mj5\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-config-out\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.344870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.345307 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.345307 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.344921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.345435 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.345373    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.346046 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.345727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.346648 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.346547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.348169 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.348121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-web-config\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.348501 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.348479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-config-volume\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.348681 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.348659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.349101 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.349080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.349192 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.349087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.349906 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.349876    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.350392 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.350366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.350667 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.350647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-config-out\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.351003 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.350982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.361124 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.361095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mj5\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-kube-api-access-g4mj5\") pod \"alertmanager-main-0\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.482715 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.482680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:41:16.814739 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:16.814709    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:41:17.049911 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:17.049870    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23937a2_09b1_47ee_a566_991f58864ab2.slice/crio-00d1fe27eca38f116252ece24fa475de0eedb77559cd6dd6aac5dc8fba80a23e WatchSource:0}: Error finding container 00d1fe27eca38f116252ece24fa475de0eedb77559cd6dd6aac5dc8fba80a23e: Status 404 returned error can't find the container with id 00d1fe27eca38f116252ece24fa475de0eedb77559cd6dd6aac5dc8fba80a23e
Sep 16 18:41:17.233978 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:17.233122    2561 generic.go:356] "Generic (PLEG): container finished" podID="4701e0a7-c118-47b8-9ff7-d38c38a7794d" containerID="75ffcc6ca594ad44037e1c5a01889594e887ae40e80400f4b11e2c85e69710f7" exitCode=0
Sep 16 18:41:17.233978 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:17.233271    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h95tc" event={"ID":"4701e0a7-c118-47b8-9ff7-d38c38a7794d","Type":"ContainerDied","Data":"75ffcc6ca594ad44037e1c5a01889594e887ae40e80400f4b11e2c85e69710f7"}
Sep 16 18:41:17.238648 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:17.238287    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz" event={"ID":"5a1a36bd-4778-4ef6-8bc8-6894080ed305","Type":"ContainerStarted","Data":"056d0e635e06ce31adb06b4f7465428709e906a9c6a858a2953ad76d2ee4fb31"}
Sep 16 18:41:17.238648 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:17.238373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz" event={"ID":"5a1a36bd-4778-4ef6-8bc8-6894080ed305","Type":"ContainerStarted","Data":"055ab723e5579c1a0cd3d5b0154d1e71be7bb9b5199ca5e2ec4b5ac2ee7e1e2a"}
Sep 16 18:41:17.240492 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:17.240462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerStarted","Data":"00d1fe27eca38f116252ece24fa475de0eedb77559cd6dd6aac5dc8fba80a23e"}
Sep 16 18:41:17.996708 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:17.996674    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"]
Sep 16 18:41:18.000756 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.000734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.010145 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:18.010087    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret"
Sep 16 18:41:18.010332 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:18.010289    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"thanos-querier-76d8d5f8d4-vhcbs\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" podUID="c8c40772-65e2-48bf-be54-71ce2842fc50" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.010600 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:18.010548    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-dockercfg-qx54n\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qx54n\"" type="*v1.Secret"
Sep 16 18:41:18.012890 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:18.012855    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-kube-rbac-proxy-web\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret"
Sep 16 18:41:18.012991 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:18.012954    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-kube-rbac-proxy-rules\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret"
Sep 16 18:41:18.013086 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:18.013061    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-tls\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret"
Sep 16 18:41:18.013139 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:18.013088    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-grpc-tls-egofqhg6uenm5\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-egofqhg6uenm5\"" type="*v1.Secret"
Sep 16 18:41:18.013324 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:18.013299    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-kube-rbac-proxy-metrics\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret"
Sep 16 18:41:18.080182 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.080143    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"]
Sep 16 18:41:18.152450 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.152420    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:41:18.157268 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.157239    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.157376 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.157274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.157376 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.157297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-grpc-tls\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.157466 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.157408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-tls\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.157466 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.157450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.157537 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.157474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.157537 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.157493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987jc\" (UniqueName: \"kubernetes.io/projected/c8c40772-65e2-48bf-be54-71ce2842fc50-kube-api-access-987jc\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.157537 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.157511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8c40772-65e2-48bf-be54-71ce2842fc50-metrics-client-ca\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.246398 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.246365    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz" event={"ID":"5a1a36bd-4778-4ef6-8bc8-6894080ed305","Type":"ContainerStarted","Data":"68b13c469adbdb78b5d615dd7228bee77b2acf8e64e79ecf8579736b51b80e28"}
Sep 16 18:41:18.248761 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.248704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h95tc" event={"ID":"4701e0a7-c118-47b8-9ff7-d38c38a7794d","Type":"ContainerStarted","Data":"2898381ddfb3f5baa4f5a84673ac72f42a2e5ebcf5f20ceb7062a77737f2510d"}
Sep 16 18:41:18.248761 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.248749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h95tc" event={"ID":"4701e0a7-c118-47b8-9ff7-d38c38a7794d","Type":"ContainerStarted","Data":"daad78bd5f49388427d56c02af43cdc60d431aec165e4fac763ac50bd00729a6"}
Sep 16 18:41:18.259007 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.258944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.259007 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.258998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987jc\" (UniqueName: \"kubernetes.io/projected/c8c40772-65e2-48bf-be54-71ce2842fc50-kube-api-access-987jc\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.259173 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.259048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8c40772-65e2-48bf-be54-71ce2842fc50-metrics-client-ca\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.259304 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.259277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.259420 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.259320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.259420 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.259387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-grpc-tls\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.259985 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.259551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-tls\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.259985 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.259629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.260441 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.260416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8c40772-65e2-48bf-be54-71ce2842fc50-metrics-client-ca\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.298458 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.298419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-987jc\" (UniqueName: \"kubernetes.io/projected/c8c40772-65e2-48bf-be54-71ce2842fc50-kube-api-access-987jc\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:18.383436 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.383377    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h95tc" podStartSLOduration=3.355336628 podStartE2EDuration="4.383359516s" totalImagesPullingTime="961.971478ms" totalInitContainerRuntime="66.05141ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:18.329812778 +0000 UTC m=+120.118141161" watchObservedRunningTime="2026-09-16 18:41:18.383359516 +0000 UTC m=+120.171687882"
Sep 16 18:41:18.384070 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:18.384009    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-hc2fz" podStartSLOduration=2.859043738 podStartE2EDuration="4.384000278s" totalImagesPullingTime="1.52495654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:18.383181646 +0000 UTC m=+120.171510011" watchObservedRunningTime="2026-09-16 18:41:18.384000278 +0000 UTC m=+120.172328668"
Sep 16 18:41:19.037809 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.037770    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-egofqhg6uenm5\""
Sep 16 18:41:19.042123 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.042098    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-grpc-tls\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.052618 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.052594    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qx54n\""
Sep 16 18:41:19.193946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.193917    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:41:19.201956 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.201929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.230547 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.230511    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"]
Sep 16 18:41:19.236312 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.236286    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:19.239648 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.239625    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tcks4\""
Sep 16 18:41:19.240259 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.240239    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:41:19.254208 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.254180    2561 generic.go:356] "Generic (PLEG): container finished" podID="d23937a2-09b1-47ee-a566-991f58864ab2" containerID="c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10" exitCode=0
Sep 16 18:41:19.254537 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.254256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerDied","Data":"c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10"}
Sep 16 18:41:19.259879 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:19.259860    2561 secret.go:189] Couldn't get secret openshift-monitoring/thanos-querier-kube-rbac-proxy: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:41:19.259879 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:19.259860    2561 secret.go:189] Couldn't get secret openshift-monitoring/thanos-querier-tls: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:41:19.260000 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:19.259878    2561 secret.go:189] Couldn't get secret openshift-monitoring/thanos-querier-kube-rbac-proxy-metrics: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:41:19.260000 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:19.259904    2561 secret.go:189] Couldn't get secret openshift-monitoring/thanos-querier-kube-rbac-proxy-rules: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:41:19.260000 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:19.259936    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy podName:c8c40772-65e2-48bf-be54-71ce2842fc50 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.759919013 +0000 UTC m=+121.548247356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-thanos-querier-kube-rbac-proxy" (UniqueName: "kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy") pod "thanos-querier-76d8d5f8d4-vhcbs" (UID: "c8c40772-65e2-48bf-be54-71ce2842fc50") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:41:19.260000 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:19.259956    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-metrics podName:c8c40772-65e2-48bf-be54-71ce2842fc50 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.759949656 +0000 UTC m=+121.548278000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-thanos-querier-kube-rbac-proxy-metrics" (UniqueName: "kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-metrics") pod "thanos-querier-76d8d5f8d4-vhcbs" (UID: "c8c40772-65e2-48bf-be54-71ce2842fc50") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:41:19.260000 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:19.259968    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-tls podName:c8c40772-65e2-48bf-be54-71ce2842fc50 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.759961616 +0000 UTC m=+121.548289961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-thanos-querier-tls" (UniqueName: "kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-tls") pod "thanos-querier-76d8d5f8d4-vhcbs" (UID: "c8c40772-65e2-48bf-be54-71ce2842fc50") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:41:19.260000 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:19.259979    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-rules podName:c8c40772-65e2-48bf-be54-71ce2842fc50 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.759974186 +0000 UTC m=+121.548302529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-thanos-querier-kube-rbac-proxy-rules" (UniqueName: "kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-rules") pod "thanos-querier-76d8d5f8d4-vhcbs" (UID: "c8c40772-65e2-48bf-be54-71ce2842fc50") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:41:19.296478 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.296401    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:41:19.301013 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.300984    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"]
Sep 16 18:41:19.371015 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.370945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twj2w\" (UniqueName: \"kubernetes.io/projected/b5bf780c-04ef-48e4-9fee-aad7d925af23-kube-api-access-twj2w\") pod \"monitoring-plugin-5d85778b9d-hp5qq\" (UID: \"b5bf780c-04ef-48e4-9fee-aad7d925af23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:19.371210 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.371117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5bf780c-04ef-48e4-9fee-aad7d925af23-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hp5qq\" (UID: \"b5bf780c-04ef-48e4-9fee-aad7d925af23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:19.472619 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.472586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twj2w\" (UniqueName: \"kubernetes.io/projected/b5bf780c-04ef-48e4-9fee-aad7d925af23-kube-api-access-twj2w\") pod \"monitoring-plugin-5d85778b9d-hp5qq\" (UID: \"b5bf780c-04ef-48e4-9fee-aad7d925af23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:19.472619 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.472625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5bf780c-04ef-48e4-9fee-aad7d925af23-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hp5qq\" (UID: \"b5bf780c-04ef-48e4-9fee-aad7d925af23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:19.475220 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.475194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5bf780c-04ef-48e4-9fee-aad7d925af23-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hp5qq\" (UID: \"b5bf780c-04ef-48e4-9fee-aad7d925af23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:19.498811 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.498771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twj2w\" (UniqueName: \"kubernetes.io/projected/b5bf780c-04ef-48e4-9fee-aad7d925af23-kube-api-access-twj2w\") pod \"monitoring-plugin-5d85778b9d-hp5qq\" (UID: \"b5bf780c-04ef-48e4-9fee-aad7d925af23\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:19.532530 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.532500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:41:19.545889 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.545855    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:19.564615 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.564590    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:41:19.592665 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.592635    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:41:19.629330 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.627046    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5455b4cb7d-64f7h"]
Sep 16 18:41:19.663512 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.663473    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5455b4cb7d-64f7h"]
Sep 16 18:41:19.663694 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.663618    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.667923 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.667893    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:41:19.668495 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.668471    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:41:19.668651 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.668538    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:41:19.668651 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.668611    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ca4h2ui3r1gku\""
Sep 16 18:41:19.668822 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.668806    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wrkc5\""
Sep 16 18:41:19.671198 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.671178    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:41:19.775716 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.775680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f926833-0910-4be6-a32c-6ffb709b8041-metrics-server-audit-profiles\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.775716 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.775716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f926833-0910-4be6-a32c-6ffb709b8041-client-ca-bundle\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.775897 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.775753    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-tls\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.775897 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.775777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f926833-0910-4be6-a32c-6ffb709b8041-audit-log\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.775897 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.775803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.775897 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.775836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.775897 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.775878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f926833-0910-4be6-a32c-6ffb709b8041-secret-metrics-server-client-certs\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.776106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.775950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.776106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.775979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sknlw\" (UniqueName: \"kubernetes.io/projected/5f926833-0910-4be6-a32c-6ffb709b8041-kube-api-access-sknlw\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.776106 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.776065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f926833-0910-4be6-a32c-6ffb709b8041-secret-metrics-server-tls\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.776224 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.776127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f926833-0910-4be6-a32c-6ffb709b8041-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.778332 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.778305    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.778453 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.778337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.778453 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.778349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.778522 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.778464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8c40772-65e2-48bf-be54-71ce2842fc50-secret-thanos-querier-tls\") pod \"thanos-querier-76d8d5f8d4-vhcbs\" (UID: \"c8c40772-65e2-48bf-be54-71ce2842fc50\") " pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.812578 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.812497    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:19.844499 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.844464    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"]
Sep 16 18:41:19.845473 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:19.845444    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bf780c_04ef_48e4_9fee_aad7d925af23.slice/crio-365959ba1f58887df001c57897b1db60812f39efda5165008ccb87ad8d7a6b02 WatchSource:0}: Error finding container 365959ba1f58887df001c57897b1db60812f39efda5165008ccb87ad8d7a6b02: Status 404 returned error can't find the container with id 365959ba1f58887df001c57897b1db60812f39efda5165008ccb87ad8d7a6b02
Sep 16 18:41:19.876924 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.876894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sknlw\" (UniqueName: \"kubernetes.io/projected/5f926833-0910-4be6-a32c-6ffb709b8041-kube-api-access-sknlw\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.876924 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.876936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f926833-0910-4be6-a32c-6ffb709b8041-secret-metrics-server-tls\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.877145 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.876956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f926833-0910-4be6-a32c-6ffb709b8041-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.877145 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.877030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f926833-0910-4be6-a32c-6ffb709b8041-metrics-server-audit-profiles\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.877145 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.877050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f926833-0910-4be6-a32c-6ffb709b8041-client-ca-bundle\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.877145 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.877089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f926833-0910-4be6-a32c-6ffb709b8041-audit-log\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.877368 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.877154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f926833-0910-4be6-a32c-6ffb709b8041-secret-metrics-server-client-certs\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.877626 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.877597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f926833-0910-4be6-a32c-6ffb709b8041-audit-log\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.878339 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.878279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f926833-0910-4be6-a32c-6ffb709b8041-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.879004 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.878679    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f926833-0910-4be6-a32c-6ffb709b8041-metrics-server-audit-profiles\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.880099 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.880065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f926833-0910-4be6-a32c-6ffb709b8041-secret-metrics-server-client-certs\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.880181 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.880153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f926833-0910-4be6-a32c-6ffb709b8041-secret-metrics-server-tls\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.880261 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.880238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f926833-0910-4be6-a32c-6ffb709b8041-client-ca-bundle\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.887523 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.887485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sknlw\" (UniqueName: \"kubernetes.io/projected/5f926833-0910-4be6-a32c-6ffb709b8041-kube-api-access-sknlw\") pod \"metrics-server-5455b4cb7d-64f7h\" (UID: \"5f926833-0910-4be6-a32c-6ffb709b8041\") " pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:19.973247 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:19.973206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:20.124736 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:20.124710    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"]
Sep 16 18:41:20.126411 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:20.126384    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c40772_65e2_48bf_be54_71ce2842fc50.slice/crio-14e966c5095b5eaa353283e89e8c2b0f0565a15bc331830116f63edc3b16ea93 WatchSource:0}: Error finding container 14e966c5095b5eaa353283e89e8c2b0f0565a15bc331830116f63edc3b16ea93: Status 404 returned error can't find the container with id 14e966c5095b5eaa353283e89e8c2b0f0565a15bc331830116f63edc3b16ea93
Sep 16 18:41:20.259874 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:20.259837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs" event={"ID":"c8c40772-65e2-48bf-be54-71ce2842fc50","Type":"ContainerStarted","Data":"14e966c5095b5eaa353283e89e8c2b0f0565a15bc331830116f63edc3b16ea93"}
Sep 16 18:41:20.261067 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:20.261039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq" event={"ID":"b5bf780c-04ef-48e4-9fee-aad7d925af23","Type":"ContainerStarted","Data":"365959ba1f58887df001c57897b1db60812f39efda5165008ccb87ad8d7a6b02"}
Sep 16 18:41:20.291313 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:20.291275    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5455b4cb7d-64f7h"]
Sep 16 18:41:20.295905 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:20.295871    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f926833_0910_4be6_a32c_6ffb709b8041.slice/crio-c7e0ee954148898dd3ea0c494851b94e6ce28bbd296d02e8dab2e66594ab102c WatchSource:0}: Error finding container c7e0ee954148898dd3ea0c494851b94e6ce28bbd296d02e8dab2e66594ab102c: Status 404 returned error can't find the container with id c7e0ee954148898dd3ea0c494851b94e6ce28bbd296d02e8dab2e66594ab102c
Sep 16 18:41:21.268313 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.268274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h" event={"ID":"5f926833-0910-4be6-a32c-6ffb709b8041","Type":"ContainerStarted","Data":"c7e0ee954148898dd3ea0c494851b94e6ce28bbd296d02e8dab2e66594ab102c"}
Sep 16 18:41:21.616670 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.615926    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:41:21.620536 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.620396    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.630985 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.630960    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:41:21.631205 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.631160    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:41:21.631205 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.631185    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:41:21.631388 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.631227    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8sz2t\""
Sep 16 18:41:21.631388 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.631167    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:41:21.631496 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.631427    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:41:21.631496 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.631476    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:41:21.632841 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.632699    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:41:21.632841 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.632813    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:41:21.633009 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.632890    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:41:21.633009 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.632995    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:41:21.633108 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.633058    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:41:21.635933 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.635910    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:41:21.636381 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.636129    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-vt9vnd8u0i9e\""
Sep 16 18:41:21.641512 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.641093    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:41:21.645623 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.645604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:41:21.664702 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.664662    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:41:21.743738 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.743711    2561 scope.go:117] "RemoveContainer" containerID="892ddfac8d704d9c3506030db54fb40123e5ad7c631c60f5edb730fb0ffc5dab"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796151    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.796557 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796491    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.797349 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.797349 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbcm\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-kube-api-access-7zbcm\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.797349 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.797349 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.797349 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-config\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.797349 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.796781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.898118 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.898118 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.898118 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.898972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbcm\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-kube-api-access-7zbcm\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.899037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899048 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.899064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899880 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.899102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-config\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.899880 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.899127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.902192 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.900707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.903905 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.903331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.903905 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.903473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.903905 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.903843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-config\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.906260 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.904538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.906260 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.904733    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.906260 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.905020    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.906260 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.905038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.906260 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.905178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.906260 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.905434    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.906260 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.905492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.906260 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.905692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.906260 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.906043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.909996 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.906751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.909996 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.907474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.909996 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.908451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.909996 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.908875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.911625 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.910684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.914026 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.913990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbcm\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-kube-api-access-7zbcm\") pod \"prometheus-k8s-0\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:21.935992 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:21.935953    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:22.275283 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.275251    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:41:22.275717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.275351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" event={"ID":"75109d51-2bb5-4be7-816d-3c4d5b5f9a5d","Type":"ContainerStarted","Data":"e8a1294fc95a1ee59bbbc2852306d913f5c418c4bb41321ac448af0bcc30c12e"}
Sep 16 18:41:22.275864 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.275834    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:41:22.278146 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.278121    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerStarted","Data":"236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320"}
Sep 16 18:41:22.278257 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.278221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerStarted","Data":"bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5"}
Sep 16 18:41:22.279776 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.279754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq" event={"ID":"b5bf780c-04ef-48e4-9fee-aad7d925af23","Type":"ContainerStarted","Data":"b0ca878b0245633785263544d8e9297fa92353f6e6a215eb59440b1ad70c7bc9"}
Sep 16 18:41:22.279988 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.279970    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:22.284790 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.284768    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq"
Sep 16 18:41:22.304252 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.304203    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz" podStartSLOduration=52.476067138 podStartE2EDuration="54.304187081s" totalImagesPullingTime="1.828119943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:22.302741952 +0000 UTC m=+124.091070317" watchObservedRunningTime="2026-09-16 18:41:22.304187081 +0000 UTC m=+124.092515448"
Sep 16 18:41:22.327351 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.327303    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hp5qq" podStartSLOduration=1.531442116 podStartE2EDuration="3.327285317s" totalImagesPullingTime="1.795843201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:22.326274917 +0000 UTC m=+124.114603284" watchObservedRunningTime="2026-09-16 18:41:22.327285317 +0000 UTC m=+124.115613682"
Sep 16 18:41:22.700980 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:22.700895    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-j9cpz"
Sep 16 18:41:23.240032 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.240000    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:41:23.249450 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:23.249426    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb3f5c9_cb92_4312_849a_289cea13579e.slice/crio-1ef6125d432ecb2d1a0ac14aad848f49d802583ebd257b27ff45700d4d2f141e WatchSource:0}: Error finding container 1ef6125d432ecb2d1a0ac14aad848f49d802583ebd257b27ff45700d4d2f141e: Status 404 returned error can't find the container with id 1ef6125d432ecb2d1a0ac14aad848f49d802583ebd257b27ff45700d4d2f141e
Sep 16 18:41:23.285006 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.284973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerStarted","Data":"1ef6125d432ecb2d1a0ac14aad848f49d802583ebd257b27ff45700d4d2f141e"}
Sep 16 18:41:23.288624 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.288586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerStarted","Data":"9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4"}
Sep 16 18:41:23.288744 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.288680    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerStarted","Data":"177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d"}
Sep 16 18:41:23.288744 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.288728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerStarted","Data":"8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6"}
Sep 16 18:41:23.290294 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.290268    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h" event={"ID":"5f926833-0910-4be6-a32c-6ffb709b8041","Type":"ContainerStarted","Data":"6475a890f279b392ab347ab4f62592a8d1645310afd0722cc271e85f1713c843"}
Sep 16 18:41:23.292258 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.292233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs" event={"ID":"c8c40772-65e2-48bf-be54-71ce2842fc50","Type":"ContainerStarted","Data":"ff55c5a4e94d065aeb70c01bffdb12c8f22f498af550572fab484ecdec98d424"}
Sep 16 18:41:23.292359 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.292295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs" event={"ID":"c8c40772-65e2-48bf-be54-71ce2842fc50","Type":"ContainerStarted","Data":"0387e50cd2a1aefeb8ec9fb4b2cf0dbe747440293383899ada655039b5a1de4e"}
Sep 16 18:41:23.292359 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.292338    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs" event={"ID":"c8c40772-65e2-48bf-be54-71ce2842fc50","Type":"ContainerStarted","Data":"db08550b2b939ef09dd90623ca195322ed46f4624390dddfbc829a4ba8a29a99"}
Sep 16 18:41:23.343596 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:23.343520    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h" podStartSLOduration=1.710824288 podStartE2EDuration="4.343496133s" totalImagesPullingTime="2.632671845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:23.338649139 +0000 UTC m=+125.126977505" watchObservedRunningTime="2026-09-16 18:41:23.343496133 +0000 UTC m=+125.131824501"
Sep 16 18:41:24.300983 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:24.300925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerStarted","Data":"69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377"}
Sep 16 18:41:24.304085 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:24.303913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs" event={"ID":"c8c40772-65e2-48bf-be54-71ce2842fc50","Type":"ContainerStarted","Data":"e5ff0875eb7bae7685666a2f3ba69504190190a40d716c8ee408d69e788b5a11"}
Sep 16 18:41:24.304085 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:24.303994    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs" event={"ID":"c8c40772-65e2-48bf-be54-71ce2842fc50","Type":"ContainerStarted","Data":"cfb8272b26e98e6b683fd048f695b11437079686033ca14836fff21dc64b262a"}
Sep 16 18:41:24.306752 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:24.306718    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerID="17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582" exitCode=0
Sep 16 18:41:24.306881 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:24.306812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerDied","Data":"17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582"}
Sep 16 18:41:24.454655 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:24.454489    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.9514312499999997 podStartE2EDuration="8.454404866s" totalImagesPullingTime="4.486332992s" totalInitContainerRuntime="16.640624ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:24.392080673 +0000 UTC m=+126.180409040" watchObservedRunningTime="2026-09-16 18:41:24.454404866 +0000 UTC m=+126.242733249"
Sep 16 18:41:25.318386 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:25.318343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs" event={"ID":"c8c40772-65e2-48bf-be54-71ce2842fc50","Type":"ContainerStarted","Data":"0f8d937b0b06e925c63e740fb5410bfbe75f8e628cc72dfd829b6cd5bd4c521f"}
Sep 16 18:41:25.318877 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:25.318505    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:25.380277 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:25.380203    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs" podStartSLOduration=4.613096971 podStartE2EDuration="8.3801809s" totalImagesPullingTime="3.767083929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:25.378361379 +0000 UTC m=+127.166689745" watchObservedRunningTime="2026-09-16 18:41:25.3801809 +0000 UTC m=+127.168509267"
Sep 16 18:41:28.335851 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.335815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerStarted","Data":"ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da"}
Sep 16 18:41:28.336242 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.335881    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerStarted","Data":"dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb"}
Sep 16 18:41:28.336242 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.335909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerStarted","Data":"2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58"}
Sep 16 18:41:28.336242 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.335977    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerStarted","Data":"77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d"}
Sep 16 18:41:28.336242 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.336003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerStarted","Data":"df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315"}
Sep 16 18:41:28.336242 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.336028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerStarted","Data":"841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c"}
Sep 16 18:41:28.399973 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.399909    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.095225757 podStartE2EDuration="7.399883139s" totalImagesPullingTime="3.284975353s" totalInitContainerRuntime="19.682029ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:28.396074793 +0000 UTC m=+130.184403159" watchObservedRunningTime="2026-09-16 18:41:28.399883139 +0000 UTC m=+130.188211506"
Sep 16 18:41:28.466029 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.465986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:41:28.468276 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.468255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08562f6-72fa-408c-9d9f-43ce0d4ced85-metrics-certs\") pod \"network-metrics-daemon-ds74d\" (UID: \"e08562f6-72fa-408c-9d9f-43ce0d4ced85\") " pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:41:28.473342 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.473317    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcswh\""
Sep 16 18:41:28.479368 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.479346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ds74d"
Sep 16 18:41:28.771632 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.771593    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7df699999f-kr6dt"]
Sep 16 18:41:28.779598 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:28.779546    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ds74d"]
Sep 16 18:41:28.781308 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:41:28.781277    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08562f6_72fa_408c_9d9f_43ce0d4ced85.slice/crio-2f0742414f12e492fcb04551b26bf08e4f972b16070ec2421bf58c339daf9104 WatchSource:0}: Error finding container 2f0742414f12e492fcb04551b26bf08e4f972b16070ec2421bf58c339daf9104: Status 404 returned error can't find the container with id 2f0742414f12e492fcb04551b26bf08e4f972b16070ec2421bf58c339daf9104
Sep 16 18:41:29.341385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:29.341350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ds74d" event={"ID":"e08562f6-72fa-408c-9d9f-43ce0d4ced85","Type":"ContainerStarted","Data":"2f0742414f12e492fcb04551b26bf08e4f972b16070ec2421bf58c339daf9104"}
Sep 16 18:41:30.347911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:30.347873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ds74d" event={"ID":"e08562f6-72fa-408c-9d9f-43ce0d4ced85","Type":"ContainerStarted","Data":"092887ad41b1905ab4fbd1d1490e55ca972c16c56f2c650c70bfed950636ec83"}
Sep 16 18:41:30.348347 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:30.347935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ds74d" event={"ID":"e08562f6-72fa-408c-9d9f-43ce0d4ced85","Type":"ContainerStarted","Data":"676892093eede62a99a2d60b3c3b99c4e7f040be05c86e7cf834b3f8f8455118"}
Sep 16 18:41:31.329645 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:31.329618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76d8d5f8d4-vhcbs"
Sep 16 18:41:31.400289 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:31.400236    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ds74d" podStartSLOduration=132.604786852 podStartE2EDuration="2m13.400217935s" totalImagesPullingTime="795.431083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:30.611483343 +0000 UTC m=+132.399811709" watchObservedRunningTime="2026-09-16 18:41:31.400217935 +0000 UTC m=+133.188546338"
Sep 16 18:41:31.936634 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:31.936600    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:41:39.974181 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:39.974128    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:39.974698 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:39.974219    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:53.792554 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:53.792493    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7df699999f-kr6dt" podUID="5ab8765f-7aac-443f-a9d5-c077f377f82c" containerName="registry" containerID="cri-o://188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d" gracePeriod=30
Sep 16 18:41:54.042881 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.042813    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:41:54.090807 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.090770    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-installation-pull-secrets\") pod \"5ab8765f-7aac-443f-a9d5-c077f377f82c\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") "
Sep 16 18:41:54.090995 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.090835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-certificates\") pod \"5ab8765f-7aac-443f-a9d5-c077f377f82c\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") "
Sep 16 18:41:54.090995 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.090867    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtckk\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-kube-api-access-qtckk\") pod \"5ab8765f-7aac-443f-a9d5-c077f377f82c\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") "
Sep 16 18:41:54.091110 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.091033    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ab8765f-7aac-443f-a9d5-c077f377f82c-ca-trust-extracted\") pod \"5ab8765f-7aac-443f-a9d5-c077f377f82c\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") "
Sep 16 18:41:54.091110 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.091073    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-bound-sa-token\") pod \"5ab8765f-7aac-443f-a9d5-c077f377f82c\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") "
Sep 16 18:41:54.091110 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.091100    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-image-registry-private-configuration\") pod \"5ab8765f-7aac-443f-a9d5-c077f377f82c\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") "
Sep 16 18:41:54.091270 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.091216    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-trusted-ca\") pod \"5ab8765f-7aac-443f-a9d5-c077f377f82c\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") "
Sep 16 18:41:54.091270 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.091260    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") pod \"5ab8765f-7aac-443f-a9d5-c077f377f82c\" (UID: \"5ab8765f-7aac-443f-a9d5-c077f377f82c\") "
Sep 16 18:41:54.091739 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.091367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5ab8765f-7aac-443f-a9d5-c077f377f82c" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:54.091739 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.091703    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-certificates\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:41:54.091953 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.091901    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5ab8765f-7aac-443f-a9d5-c077f377f82c" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:54.093429 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.093389    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5ab8765f-7aac-443f-a9d5-c077f377f82c" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:41:54.093606 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.093503    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5ab8765f-7aac-443f-a9d5-c077f377f82c" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:41:54.093685 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.093623    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5ab8765f-7aac-443f-a9d5-c077f377f82c" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:54.093743 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.093713    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-kube-api-access-qtckk" (OuterVolumeSpecName: "kube-api-access-qtckk") pod "5ab8765f-7aac-443f-a9d5-c077f377f82c" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c"). InnerVolumeSpecName "kube-api-access-qtckk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:54.093783 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.093749    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5ab8765f-7aac-443f-a9d5-c077f377f82c" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:54.099682 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.099641    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8765f-7aac-443f-a9d5-c077f377f82c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5ab8765f-7aac-443f-a9d5-c077f377f82c" (UID: "5ab8765f-7aac-443f-a9d5-c077f377f82c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:54.192670 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.192630    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-registry-tls\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:41:54.192670 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.192661    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-installation-pull-secrets\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:41:54.192670 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.192672    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtckk\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-kube-api-access-qtckk\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:41:54.192670 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.192680    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ab8765f-7aac-443f-a9d5-c077f377f82c-ca-trust-extracted\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:41:54.192933 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.192690    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ab8765f-7aac-443f-a9d5-c077f377f82c-bound-sa-token\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:41:54.192933 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.192701    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ab8765f-7aac-443f-a9d5-c077f377f82c-image-registry-private-configuration\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:41:54.192933 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.192709    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ab8765f-7aac-443f-a9d5-c077f377f82c-trusted-ca\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:41:54.447760 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.447670    2561 generic.go:356] "Generic (PLEG): container finished" podID="5ab8765f-7aac-443f-a9d5-c077f377f82c" containerID="188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d" exitCode=0
Sep 16 18:41:54.447760 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.447707    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df699999f-kr6dt"
Sep 16 18:41:54.447760 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.447718    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df699999f-kr6dt" event={"ID":"5ab8765f-7aac-443f-a9d5-c077f377f82c","Type":"ContainerDied","Data":"188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d"}
Sep 16 18:41:54.447760 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.447759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df699999f-kr6dt" event={"ID":"5ab8765f-7aac-443f-a9d5-c077f377f82c","Type":"ContainerDied","Data":"239ed28d34eb70d13b9b2fbf47385155db91aebdab65cb47079da049563b3632"}
Sep 16 18:41:54.448078 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.447777    2561 scope.go:117] "RemoveContainer" containerID="188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d"
Sep 16 18:41:54.458598 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.458555    2561 scope.go:117] "RemoveContainer" containerID="188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d"
Sep 16 18:41:54.458956 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:41:54.458934    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d\": container with ID starting with 188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d not found: ID does not exist" containerID="188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d"
Sep 16 18:41:54.459072 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.458972    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d"} err="failed to get container status \"188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d\": rpc error: code = NotFound desc = could not find container \"188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d\": container with ID starting with 188d6f87d8635d9778422d7cc5f7a64f3fab6baa3c11f72945a71229617d773d not found: ID does not exist"
Sep 16 18:41:54.479826 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.479795    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7df699999f-kr6dt"]
Sep 16 18:41:54.490978 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.490933    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7df699999f-kr6dt"]
Sep 16 18:41:54.748871 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:54.748829    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ab8765f-7aac-443f-a9d5-c077f377f82c" path="/var/lib/kubelet/pods/5ab8765f-7aac-443f-a9d5-c077f377f82c/volumes"
Sep 16 18:41:59.979135 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:59.979105    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:41:59.982896 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:41:59.982875    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5455b4cb7d-64f7h"
Sep 16 18:42:21.936752 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:21.936713    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.955035 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:21.955005    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.599688 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:22.599658    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:35.837293 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:35.836020    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:35.837293 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:35.836544    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="alertmanager" containerID="cri-o://bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5" gracePeriod=120
Sep 16 18:42:35.837293 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:35.836970    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="config-reloader" containerID="cri-o://236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320" gracePeriod=120
Sep 16 18:42:35.837293 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:35.837152    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy-web" containerID="cri-o://8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6" gracePeriod=120
Sep 16 18:42:35.838608 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:35.837461    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy" containerID="cri-o://177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d" gracePeriod=120
Sep 16 18:42:35.838608 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:35.837521    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="prom-label-proxy" containerID="cri-o://69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377" gracePeriod=120
Sep 16 18:42:35.838608 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:35.837667    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy-metric" containerID="cri-o://9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4" gracePeriod=120
Sep 16 18:42:36.640229 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640189    2561 generic.go:356] "Generic (PLEG): container finished" podID="d23937a2-09b1-47ee-a566-991f58864ab2" containerID="69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377" exitCode=0
Sep 16 18:42:36.640229 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640217    2561 generic.go:356] "Generic (PLEG): container finished" podID="d23937a2-09b1-47ee-a566-991f58864ab2" containerID="9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4" exitCode=0
Sep 16 18:42:36.640229 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640223    2561 generic.go:356] "Generic (PLEG): container finished" podID="d23937a2-09b1-47ee-a566-991f58864ab2" containerID="177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d" exitCode=0
Sep 16 18:42:36.640229 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640229    2561 generic.go:356] "Generic (PLEG): container finished" podID="d23937a2-09b1-47ee-a566-991f58864ab2" containerID="236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320" exitCode=0
Sep 16 18:42:36.640229 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640234    2561 generic.go:356] "Generic (PLEG): container finished" podID="d23937a2-09b1-47ee-a566-991f58864ab2" containerID="bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5" exitCode=0
Sep 16 18:42:36.640515 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerDied","Data":"69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377"}
Sep 16 18:42:36.640515 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640322    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerDied","Data":"9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4"}
Sep 16 18:42:36.640515 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerDied","Data":"177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d"}
Sep 16 18:42:36.640515 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerDied","Data":"236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320"}
Sep 16 18:42:36.640515 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:36.640395    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerDied","Data":"bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5"}
Sep 16 18:42:37.085780 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.085755    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.181313 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.181226    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4mj5\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-kube-api-access-g4mj5\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.181313 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.181288    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-config-out\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.181531 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.181315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-cluster-tls-config\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.181531 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.181349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-trusted-ca-bundle\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.181531 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.181387    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.181531 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.181432    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-tls-assets\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.181531 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.181455    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-config-volume\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.181531 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.181489    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.182833 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.182034    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-main-tls\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.182833 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.182075    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-web-config\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.182833 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.182078    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:37.182833 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.182114    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-main-db\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.182833 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.182142    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-metrics-client-ca\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.182833 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.182210    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-web\") pod \"d23937a2-09b1-47ee-a566-991f58864ab2\" (UID: \"d23937a2-09b1-47ee-a566-991f58864ab2\") "
Sep 16 18:42:37.182833 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.182498    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.183331 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.182912    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:37.183331 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.182984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:37.184418 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.184381    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-kube-api-access-g4mj5" (OuterVolumeSpecName: "kube-api-access-g4mj5") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "kube-api-access-g4mj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:37.185552 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.185530    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:37.186192 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.186158    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:37.186305 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.186188    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-config-out" (OuterVolumeSpecName: "config-out") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:37.186305 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.186243    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:37.186586 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.186545    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:37.186753 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.186735    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-config-volume" (OuterVolumeSpecName: "config-volume") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:37.187055 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.187032    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:37.191071 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.191044    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:37.197874 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.197843    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-web-config" (OuterVolumeSpecName: "web-config") pod "d23937a2-09b1-47ee-a566-991f58864ab2" (UID: "d23937a2-09b1-47ee-a566-991f58864ab2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:37.283480 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283440    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283480 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283472    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-main-tls\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283504    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-web-config\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283513    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-alertmanager-main-db\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283525    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d23937a2-09b1-47ee-a566-991f58864ab2-metrics-client-ca\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283534    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283543    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4mj5\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-kube-api-access-g4mj5\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283551    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d23937a2-09b1-47ee-a566-991f58864ab2-config-out\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283560    2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-cluster-tls-config\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283594    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283604    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d23937a2-09b1-47ee-a566-991f58864ab2-tls-assets\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.283717 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.283611    2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d23937a2-09b1-47ee-a566-991f58864ab2-config-volume\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:37.648410 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.648374    2561 generic.go:356] "Generic (PLEG): container finished" podID="d23937a2-09b1-47ee-a566-991f58864ab2" containerID="8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6" exitCode=0
Sep 16 18:42:37.648602 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.648451    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.648602 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.648454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerDied","Data":"8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6"}
Sep 16 18:42:37.648602 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.648536    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d23937a2-09b1-47ee-a566-991f58864ab2","Type":"ContainerDied","Data":"00d1fe27eca38f116252ece24fa475de0eedb77559cd6dd6aac5dc8fba80a23e"}
Sep 16 18:42:37.648602 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.648594    2561 scope.go:117] "RemoveContainer" containerID="69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377"
Sep 16 18:42:37.659860 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.658719    2561 scope.go:117] "RemoveContainer" containerID="9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4"
Sep 16 18:42:37.667311 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.667279    2561 scope.go:117] "RemoveContainer" containerID="177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d"
Sep 16 18:42:37.674791 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.674770    2561 scope.go:117] "RemoveContainer" containerID="8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6"
Sep 16 18:42:37.681883 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.681859    2561 scope.go:117] "RemoveContainer" containerID="236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320"
Sep 16 18:42:37.686304 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.686280    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:37.690264 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.690242    2561 scope.go:117] "RemoveContainer" containerID="bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5"
Sep 16 18:42:37.693320 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.693297    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:37.698296 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.698275    2561 scope.go:117] "RemoveContainer" containerID="c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10"
Sep 16 18:42:37.705635 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.705611    2561 scope.go:117] "RemoveContainer" containerID="69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377"
Sep 16 18:42:37.705938 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:37.705919    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377\": container with ID starting with 69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377 not found: ID does not exist" containerID="69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377"
Sep 16 18:42:37.706019 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.705949    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377"} err="failed to get container status \"69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377\": rpc error: code = NotFound desc = could not find container \"69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377\": container with ID starting with 69c5d6e9697e4bebbe9cd0eea8357e494560405ee502a40a5137fc43a9e37377 not found: ID does not exist"
Sep 16 18:42:37.706019 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.705971    2561 scope.go:117] "RemoveContainer" containerID="9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4"
Sep 16 18:42:37.706224 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:37.706207    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4\": container with ID starting with 9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4 not found: ID does not exist" containerID="9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4"
Sep 16 18:42:37.706269 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.706228    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4"} err="failed to get container status \"9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4\": rpc error: code = NotFound desc = could not find container \"9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4\": container with ID starting with 9b8cedd68196dbc1895c59d437e456940aac7f01cd65871e4c5af3569b993cc4 not found: ID does not exist"
Sep 16 18:42:37.706269 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.706241    2561 scope.go:117] "RemoveContainer" containerID="177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d"
Sep 16 18:42:37.706441 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:37.706426    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d\": container with ID starting with 177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d not found: ID does not exist" containerID="177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d"
Sep 16 18:42:37.706516 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.706444    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d"} err="failed to get container status \"177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d\": rpc error: code = NotFound desc = could not find container \"177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d\": container with ID starting with 177ca3c0008dd3affd85d21192bf9f06b94cfbf2bdfb4caca95977b1ce5b937d not found: ID does not exist"
Sep 16 18:42:37.706516 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.706456    2561 scope.go:117] "RemoveContainer" containerID="8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6"
Sep 16 18:42:37.706723 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:37.706705    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6\": container with ID starting with 8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6 not found: ID does not exist" containerID="8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6"
Sep 16 18:42:37.706769 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.706730    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6"} err="failed to get container status \"8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6\": rpc error: code = NotFound desc = could not find container \"8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6\": container with ID starting with 8efeb2a7674fde3d6a6328ec7307f2fe9c3e6856403bf39de3d2b9221fc49da6 not found: ID does not exist"
Sep 16 18:42:37.706769 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.706746    2561 scope.go:117] "RemoveContainer" containerID="236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320"
Sep 16 18:42:37.707024 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:37.707001    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320\": container with ID starting with 236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320 not found: ID does not exist" containerID="236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320"
Sep 16 18:42:37.707110 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.707026    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320"} err="failed to get container status \"236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320\": rpc error: code = NotFound desc = could not find container \"236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320\": container with ID starting with 236afd4047450bb35f27201188f3b993195ad786dd8cb540fcfb3d3cda208320 not found: ID does not exist"
Sep 16 18:42:37.707110 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.707042    2561 scope.go:117] "RemoveContainer" containerID="bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5"
Sep 16 18:42:37.707264 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:37.707246    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5\": container with ID starting with bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5 not found: ID does not exist" containerID="bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5"
Sep 16 18:42:37.707335 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.707270    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5"} err="failed to get container status \"bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5\": rpc error: code = NotFound desc = could not find container \"bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5\": container with ID starting with bc82c8d89f9f7b1dbd80ab69bd0c462854e7c659ee961479f69e2da6bc5f9ac5 not found: ID does not exist"
Sep 16 18:42:37.707335 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.707289    2561 scope.go:117] "RemoveContainer" containerID="c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10"
Sep 16 18:42:37.707543 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:37.707527    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10\": container with ID starting with c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10 not found: ID does not exist" containerID="c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10"
Sep 16 18:42:37.707601 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:37.707546    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10"} err="failed to get container status \"c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10\": rpc error: code = NotFound desc = could not find container \"c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10\": container with ID starting with c8184d2ffb4573c7c583ac0c640d3cda6e5f0064d1e18ad0f3f7e4ac3757af10 not found: ID does not exist"
Sep 16 18:42:38.748241 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:38.748210    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" path="/var/lib/kubelet/pods/d23937a2-09b1-47ee-a566-991f58864ab2/volumes"
Sep 16 18:42:40.101092 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.101055    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:42:40.101640 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.101469    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="prometheus" containerID="cri-o://841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c" gracePeriod=600
Sep 16 18:42:40.101933 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.101900    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da" gracePeriod=600
Sep 16 18:42:40.102061 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.102033    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy" containerID="cri-o://dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb" gracePeriod=600
Sep 16 18:42:40.102168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.102128    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy-web" containerID="cri-o://2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58" gracePeriod=600
Sep 16 18:42:40.102251 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.102227    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="thanos-sidecar" containerID="cri-o://77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d" gracePeriod=600
Sep 16 18:42:40.102338 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.102317    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="config-reloader" containerID="cri-o://df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315" gracePeriod=600
Sep 16 18:42:40.663813 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.663778    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerID="ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da" exitCode=0
Sep 16 18:42:40.663813 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.663807    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerID="dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb" exitCode=0
Sep 16 18:42:40.663813 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.663817    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerID="77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d" exitCode=0
Sep 16 18:42:40.663813 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.663829    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerID="df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315" exitCode=0
Sep 16 18:42:40.664091 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.663837    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerID="841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c" exitCode=0
Sep 16 18:42:40.664091 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.663851    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerDied","Data":"ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da"}
Sep 16 18:42:40.664091 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.663950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerDied","Data":"dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb"}
Sep 16 18:42:40.664091 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.663977    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerDied","Data":"77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d"}
Sep 16 18:42:40.664091 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.664002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerDied","Data":"df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315"}
Sep 16 18:42:40.664091 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:40.664025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerDied","Data":"841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c"}
Sep 16 18:42:41.460967 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.460940    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.519183 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519142    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-grpc-tls\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519183 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-0\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519216    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-metrics-client-certs\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519233    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-web-config\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519252    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-config-out\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519276    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519295    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-kube-rbac-proxy\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-tls\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519360    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-2\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-trusted-ca-bundle\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519412    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zbcm\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-kube-api-access-7zbcm\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519438    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-config\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519466    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-serving-certs-ca-bundle\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519500    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-tls-assets\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519548    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-1\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519598    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-thanos-prometheus-http-client-file\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519642    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-metrics-client-ca\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-kubelet-serving-ca-bundle\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519738    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-db\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ccb3f5c9-cb92-4312-849a-289cea13579e\" (UID: \"ccb3f5c9-cb92-4312-849a-289cea13579e\") "
Sep 16 18:42:41.519911 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.519845    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:41.520384 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.520030    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.520384 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.520135    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:41.520551 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.520523    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:41.521327 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.521299    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:41.522581 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.522044    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:41.522581 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.522378    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-config-out" (OuterVolumeSpecName: "config-out") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:41.522774 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.522650    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:41.522774 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.522675    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:41.522892 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.522783    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:41.522892 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.522800    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:41.523059 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.523031    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:41.523161 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.523131    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:41.524089 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.524033    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:41.524278 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.524246    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:41.524363 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.524272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:41.524532 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.524504    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:41.525017 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.524990    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-kube-api-access-7zbcm" (OuterVolumeSpecName: "kube-api-access-7zbcm") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "kube-api-access-7zbcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:41.525102 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.524997    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-config" (OuterVolumeSpecName: "config") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:41.525281 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.525262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:41.533394 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.533366    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-web-config" (OuterVolumeSpecName: "web-config") pod "ccb3f5c9-cb92-4312-849a-289cea13579e" (UID: "ccb3f5c9-cb92-4312-849a-289cea13579e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:41.621444 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621394    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-metrics-client-ca\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621444 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621440    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621444 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621453    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-db\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621465    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621475    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-grpc-tls\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621483    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621492    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-metrics-client-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621500    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-web-config\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621508    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb3f5c9-cb92-4312-849a-289cea13579e-config-out\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621516    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621523    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-kube-rbac-proxy\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621531    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621539    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621547    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbcm\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-kube-api-access-7zbcm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621556    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-config\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621564    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621601    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb3f5c9-cb92-4312-849a-289cea13579e-tls-assets\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621610    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb3f5c9-cb92-4312-849a-289cea13579e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.621710 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.621618    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb3f5c9-cb92-4312-849a-289cea13579e-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:42:41.670720 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.670682    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerID="2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58" exitCode=0
Sep 16 18:42:41.670901 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.670767    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.670901 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.670762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerDied","Data":"2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58"}
Sep 16 18:42:41.671027 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.670911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb3f5c9-cb92-4312-849a-289cea13579e","Type":"ContainerDied","Data":"1ef6125d432ecb2d1a0ac14aad848f49d802583ebd257b27ff45700d4d2f141e"}
Sep 16 18:42:41.671027 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.670944    2561 scope.go:117] "RemoveContainer" containerID="ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da"
Sep 16 18:42:41.681603 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.680157    2561 scope.go:117] "RemoveContainer" containerID="dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb"
Sep 16 18:42:41.689488 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.689469    2561 scope.go:117] "RemoveContainer" containerID="2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58"
Sep 16 18:42:41.696352 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.696324    2561 scope.go:117] "RemoveContainer" containerID="77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d"
Sep 16 18:42:41.701767 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.701742    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:42:41.704561 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.704538    2561 scope.go:117] "RemoveContainer" containerID="df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315"
Sep 16 18:42:41.705643 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.705616    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:42:41.711464 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.711444    2561 scope.go:117] "RemoveContainer" containerID="841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c"
Sep 16 18:42:41.718801 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.718782    2561 scope.go:117] "RemoveContainer" containerID="17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582"
Sep 16 18:42:41.725604 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.725589    2561 scope.go:117] "RemoveContainer" containerID="ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da"
Sep 16 18:42:41.725901 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:41.725880    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da\": container with ID starting with ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da not found: ID does not exist" containerID="ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da"
Sep 16 18:42:41.725982 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.725914    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da"} err="failed to get container status \"ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da\": rpc error: code = NotFound desc = could not find container \"ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da\": container with ID starting with ab46eef55bea96dc510a96cbcf6a2199aa1feda8a01d9349cadaee48808162da not found: ID does not exist"
Sep 16 18:42:41.725982 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.725941    2561 scope.go:117] "RemoveContainer" containerID="dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb"
Sep 16 18:42:41.726225 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:41.726206    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb\": container with ID starting with dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb not found: ID does not exist" containerID="dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb"
Sep 16 18:42:41.726268 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.726231    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb"} err="failed to get container status \"dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb\": rpc error: code = NotFound desc = could not find container \"dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb\": container with ID starting with dceae664238c27cbdef82a5f1b2e2d2ae126845a6bd27e80f2eaff31dbef44cb not found: ID does not exist"
Sep 16 18:42:41.726268 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.726247    2561 scope.go:117] "RemoveContainer" containerID="2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58"
Sep 16 18:42:41.726526 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:41.726506    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58\": container with ID starting with 2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58 not found: ID does not exist" containerID="2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58"
Sep 16 18:42:41.726595 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.726533    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58"} err="failed to get container status \"2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58\": rpc error: code = NotFound desc = could not find container \"2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58\": container with ID starting with 2811439ce649136add683cf8fdb1b16a7d4e3bf45ef49ab1cebb284bbd959e58 not found: ID does not exist"
Sep 16 18:42:41.726595 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.726550    2561 scope.go:117] "RemoveContainer" containerID="77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d"
Sep 16 18:42:41.726899 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:41.726882    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d\": container with ID starting with 77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d not found: ID does not exist" containerID="77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d"
Sep 16 18:42:41.726966 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.726914    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d"} err="failed to get container status \"77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d\": rpc error: code = NotFound desc = could not find container \"77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d\": container with ID starting with 77ace7d299ba462194a196fdac97ad7309b00b5e46372b74e78e53f64934ad0d not found: ID does not exist"
Sep 16 18:42:41.726966 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.726934    2561 scope.go:117] "RemoveContainer" containerID="df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315"
Sep 16 18:42:41.727196 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:41.727176    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315\": container with ID starting with df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315 not found: ID does not exist" containerID="df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315"
Sep 16 18:42:41.727240 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.727202    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315"} err="failed to get container status \"df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315\": rpc error: code = NotFound desc = could not find container \"df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315\": container with ID starting with df585617e590091fa23b70f092bf6b8435002778177705658e57895b137fe315 not found: ID does not exist"
Sep 16 18:42:41.727240 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.727221    2561 scope.go:117] "RemoveContainer" containerID="841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c"
Sep 16 18:42:41.727449 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:41.727434    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c\": container with ID starting with 841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c not found: ID does not exist" containerID="841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c"
Sep 16 18:42:41.727493 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.727451    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c"} err="failed to get container status \"841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c\": rpc error: code = NotFound desc = could not find container \"841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c\": container with ID starting with 841e3ebd60cb421b8dd169e5cb2632ce6de055ba9107035eb19d75827aed5e7c not found: ID does not exist"
Sep 16 18:42:41.727493 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.727462    2561 scope.go:117] "RemoveContainer" containerID="17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582"
Sep 16 18:42:41.727671 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:42:41.727657    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582\": container with ID starting with 17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582 not found: ID does not exist" containerID="17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582"
Sep 16 18:42:41.727720 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.727673    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582"} err="failed to get container status \"17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582\": rpc error: code = NotFound desc = could not find container \"17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582\": container with ID starting with 17f28a9338f50970fd0e9fe521efb891a0491ce0d7229955c855feded7779582 not found: ID does not exist"
Sep 16 18:42:41.751852 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.751806    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:42:41.752143 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752129    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="prometheus"
Sep 16 18:42:41.752196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752144    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="prometheus"
Sep 16 18:42:41.752196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752154    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy-web"
Sep 16 18:42:41.752196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752159    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy-web"
Sep 16 18:42:41.752196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752168    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="alertmanager"
Sep 16 18:42:41.752196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752174    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="alertmanager"
Sep 16 18:42:41.752196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752181    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy"
Sep 16 18:42:41.752196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752186    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy"
Sep 16 18:42:41.752196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752194    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy-metric"
Sep 16 18:42:41.752196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752200    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy-metric"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752208    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="init-config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752213    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="init-config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752218    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752223    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752230    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752234    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752240    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752245    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752252    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy-thanos"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752256    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy-thanos"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752264    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="init-config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752269    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="init-config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752276    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab8765f-7aac-443f-a9d5-c077f377f82c" containerName="registry"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752281    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab8765f-7aac-443f-a9d5-c077f377f82c" containerName="registry"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752287    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy-web"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752292    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy-web"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752299    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="thanos-sidecar"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752304    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="thanos-sidecar"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752310    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="prom-label-proxy"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752315    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="prom-label-proxy"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752363    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy-metric"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752372    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752378    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="alertmanager"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752383    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy-web"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752389    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="kube-rbac-proxy"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752394    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="thanos-sidecar"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752399    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy-web"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752404    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ab8765f-7aac-443f-a9d5-c077f377f82c" containerName="registry"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752409    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752413    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="prometheus"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752418    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="kube-rbac-proxy-thanos"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752424    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" containerName="config-reloader"
Sep 16 18:42:41.752486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.752429    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23937a2-09b1-47ee-a566-991f58864ab2" containerName="prom-label-proxy"
Sep 16 18:42:41.758028 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.758007    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.763241 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.763206    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:42:41.763409 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.763250    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:42:41.764462 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.764443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:42:41.764462 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.764447    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:42:41.764722 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.764703    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-vt9vnd8u0i9e\""
Sep 16 18:42:41.767059 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.767036    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:42:41.767183 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.767154    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:42:41.767183 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.767155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:42:41.767340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.767248    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:42:41.767479 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.767463    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:42:41.767913 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.767899    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:42:41.769664 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.769621    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:42:41.770010 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.769992    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:42:41.770080 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.769996    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8sz2t\""
Sep 16 18:42:41.772153 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.772137    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:42:41.786005 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.785961    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:42:41.788325 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.786854    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:42:41.822666 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3de4adda-c570-4c43-87b8-65228d5d0a32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.822666 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.822896 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.822896 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822753    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.822896 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.822896 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.822896 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs9bv\" (UniqueName: \"kubernetes.io/projected/3de4adda-c570-4c43-87b8-65228d5d0a32-kube-api-access-gs9bv\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-web-config\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-config\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.822984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.823005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.823029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.823064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.823090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.823123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823594 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.823174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823594 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.823207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823594 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.823240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3de4adda-c570-4c43-87b8-65228d5d0a32-config-out\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.823594 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.823321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924402 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924549 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924549 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924549 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs9bv\" (UniqueName: \"kubernetes.io/projected/3de4adda-c570-4c43-87b8-65228d5d0a32-kube-api-access-gs9bv\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924549 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-web-config\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924549 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-config\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924824 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924824 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924824 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924824 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924824 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924975 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924975 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924975 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924975 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924975 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.924975 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924965    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3de4adda-c570-4c43-87b8-65228d5d0a32-config-out\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.925236 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.924996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.925236 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.925032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3de4adda-c570-4c43-87b8-65228d5d0a32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.925236 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.925055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.925236 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.925083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.925635 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.925593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.925720 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.925662    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.925870 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.925843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.926688 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.926236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.926949 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.926923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.927031 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.926932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.927989 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.927946    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-config\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.928128 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.928105    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.928304 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.928282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.928507 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.928489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3de4adda-c570-4c43-87b8-65228d5d0a32-config-out\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.929069 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.929041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-web-config\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.929337 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.929319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3de4adda-c570-4c43-87b8-65228d5d0a32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.930300 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.930268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.930300 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.930286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.930461 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.930355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.930706 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.930686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3de4adda-c570-4c43-87b8-65228d5d0a32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.930799 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.930784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.931395 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.931381    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3de4adda-c570-4c43-87b8-65228d5d0a32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:41.940414 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:41.940394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs9bv\" (UniqueName: \"kubernetes.io/projected/3de4adda-c570-4c43-87b8-65228d5d0a32-kube-api-access-gs9bv\") pod \"prometheus-k8s-0\" (UID: \"3de4adda-c570-4c43-87b8-65228d5d0a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:42.067435 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:42.067341    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:42.364903 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:42.364878    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:42:42.366869 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:42:42.366840    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de4adda_c570_4c43_87b8_65228d5d0a32.slice/crio-2ad3245bcf7c80985a79173a8d6b1ab691709b6c65755f3f3f3c50e0ca9f84ee WatchSource:0}: Error finding container 2ad3245bcf7c80985a79173a8d6b1ab691709b6c65755f3f3f3c50e0ca9f84ee: Status 404 returned error can't find the container with id 2ad3245bcf7c80985a79173a8d6b1ab691709b6c65755f3f3f3c50e0ca9f84ee
Sep 16 18:42:42.677154 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:42.677060    2561 generic.go:356] "Generic (PLEG): container finished" podID="3de4adda-c570-4c43-87b8-65228d5d0a32" containerID="31ba89387656591874de509507299209d31a0b74d631b8faa96d4726a2e994a8" exitCode=0
Sep 16 18:42:42.677591 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:42.677150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3de4adda-c570-4c43-87b8-65228d5d0a32","Type":"ContainerDied","Data":"31ba89387656591874de509507299209d31a0b74d631b8faa96d4726a2e994a8"}
Sep 16 18:42:42.677591 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:42.677232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3de4adda-c570-4c43-87b8-65228d5d0a32","Type":"ContainerStarted","Data":"2ad3245bcf7c80985a79173a8d6b1ab691709b6c65755f3f3f3c50e0ca9f84ee"}
Sep 16 18:42:42.749713 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:42.749685    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb3f5c9-cb92-4312-849a-289cea13579e" path="/var/lib/kubelet/pods/ccb3f5c9-cb92-4312-849a-289cea13579e/volumes"
Sep 16 18:42:43.686881 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:43.686796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3de4adda-c570-4c43-87b8-65228d5d0a32","Type":"ContainerStarted","Data":"274ed1e83eced0801f31058ad41f1031f305354b7589a8f4edae4a8b89528807"}
Sep 16 18:42:43.687254 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:43.686892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3de4adda-c570-4c43-87b8-65228d5d0a32","Type":"ContainerStarted","Data":"f60890511c7870386e983019ad13676a195997c82ecca720b0b5f24cd6475c1a"}
Sep 16 18:42:43.687254 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:43.686921    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3de4adda-c570-4c43-87b8-65228d5d0a32","Type":"ContainerStarted","Data":"4f4b5fed8f95defdc6f6591d96767a59e13e83abe7df8ea8bbffa20af9b128cd"}
Sep 16 18:42:43.687254 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:43.686945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3de4adda-c570-4c43-87b8-65228d5d0a32","Type":"ContainerStarted","Data":"ee5114cece075fd1f94155219f739eb74bbc8502cffd4f10b3eebc3f7682e5df"}
Sep 16 18:42:43.687254 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:43.686968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3de4adda-c570-4c43-87b8-65228d5d0a32","Type":"ContainerStarted","Data":"7fdbe08ac4e927b41e0962a4a0968dcf9073b9a1b15f04478530052973441c14"}
Sep 16 18:42:43.687254 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:43.686995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3de4adda-c570-4c43-87b8-65228d5d0a32","Type":"ContainerStarted","Data":"f6fea05d35e79809b882692f066e5abce29a23f9c4e0fe22a9d9e7ba75ad37c2"}
Sep 16 18:42:43.725727 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:43.725650    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.703641023 podStartE2EDuration="2.725621287s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.980264ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:43.723118474 +0000 UTC m=+205.511446841" watchObservedRunningTime="2026-09-16 18:42:43.725621287 +0000 UTC m=+205.513949655"
Sep 16 18:42:47.068058 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:42:47.068011    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.067938 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:43:42.067834    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.083647 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:43:42.083615    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.931372 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:43:42.931343    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:18.638453 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:44:18.638421    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:44:18.645780 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:44:18.645757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:44:18.648670 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:44:18.648649    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:46.832027 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:46.831935    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pt7kh"]
Sep 16 18:45:46.835386 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:46.835359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:46.838373 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:46.838336    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:45:46.839191 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:46.839173    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:45:46.839300 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:46.839192    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-srcch\""
Sep 16 18:45:46.907328 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:46.907294    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pt7kh"]
Sep 16 18:45:46.961863 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:46.961824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/319cad0c-f880-4db6-913f-6175cdf10b8d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pt7kh\" (UID: \"319cad0c-f880-4db6-913f-6175cdf10b8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:46.962044 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:46.961903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxmm6\" (UniqueName: \"kubernetes.io/projected/319cad0c-f880-4db6-913f-6175cdf10b8d-kube-api-access-wxmm6\") pod \"cert-manager-webhook-6486ffd997-pt7kh\" (UID: \"319cad0c-f880-4db6-913f-6175cdf10b8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:47.063066 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:47.063012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxmm6\" (UniqueName: \"kubernetes.io/projected/319cad0c-f880-4db6-913f-6175cdf10b8d-kube-api-access-wxmm6\") pod \"cert-manager-webhook-6486ffd997-pt7kh\" (UID: \"319cad0c-f880-4db6-913f-6175cdf10b8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:47.063247 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:47.063100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/319cad0c-f880-4db6-913f-6175cdf10b8d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pt7kh\" (UID: \"319cad0c-f880-4db6-913f-6175cdf10b8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:47.075748 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:47.075707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxmm6\" (UniqueName: \"kubernetes.io/projected/319cad0c-f880-4db6-913f-6175cdf10b8d-kube-api-access-wxmm6\") pod \"cert-manager-webhook-6486ffd997-pt7kh\" (UID: \"319cad0c-f880-4db6-913f-6175cdf10b8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:47.076173 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:47.076151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/319cad0c-f880-4db6-913f-6175cdf10b8d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pt7kh\" (UID: \"319cad0c-f880-4db6-913f-6175cdf10b8d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:47.156070 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:47.155970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:47.433592 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:47.433545    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pt7kh"]
Sep 16 18:45:47.434994 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:45:47.434965    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod319cad0c_f880_4db6_913f_6175cdf10b8d.slice/crio-b1f9a1010b64425fbefd7807e935130d2bf8f4472e206e8701bb1f68f8e7a82b WatchSource:0}: Error finding container b1f9a1010b64425fbefd7807e935130d2bf8f4472e206e8701bb1f68f8e7a82b: Status 404 returned error can't find the container with id b1f9a1010b64425fbefd7807e935130d2bf8f4472e206e8701bb1f68f8e7a82b
Sep 16 18:45:47.437147 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:47.437125    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:48.392887 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:48.392859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh" event={"ID":"319cad0c-f880-4db6-913f-6175cdf10b8d","Type":"ContainerStarted","Data":"b1f9a1010b64425fbefd7807e935130d2bf8f4472e206e8701bb1f68f8e7a82b"}
Sep 16 18:45:52.411280 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:52.411169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh" event={"ID":"319cad0c-f880-4db6-913f-6175cdf10b8d","Type":"ContainerStarted","Data":"3807a11bfc4322a93e82d99e2966c22973f03948f7b24a36dcb1f51cb011f4ba"}
Sep 16 18:45:52.411280 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:52.411239    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:52.436476 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:52.436283    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh" podStartSLOduration=1.8133719990000001 podStartE2EDuration="6.436261165s" totalImagesPullingTime="4.622889166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:52.434562232 +0000 UTC m=+394.222890599" watchObservedRunningTime="2026-09-16 18:45:52.436261165 +0000 UTC m=+394.224589531"
Sep 16 18:45:53.958579 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:53.958505    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-926k2"]
Sep 16 18:45:53.961863 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:53.961844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2"
Sep 16 18:45:53.964088 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:53.964059    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6zh4d\""
Sep 16 18:45:53.970991 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:53.970963    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-926k2"]
Sep 16 18:45:54.023768 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:54.023735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0815c525-5349-44d2-a535-3db6bbe9521a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-926k2\" (UID: \"0815c525-5349-44d2-a535-3db6bbe9521a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2"
Sep 16 18:45:54.023768 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:54.023782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8n6b\" (UniqueName: \"kubernetes.io/projected/0815c525-5349-44d2-a535-3db6bbe9521a-kube-api-access-s8n6b\") pod \"cert-manager-cainjector-559cdd8c87-926k2\" (UID: \"0815c525-5349-44d2-a535-3db6bbe9521a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2"
Sep 16 18:45:54.124890 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:54.124846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8n6b\" (UniqueName: \"kubernetes.io/projected/0815c525-5349-44d2-a535-3db6bbe9521a-kube-api-access-s8n6b\") pod \"cert-manager-cainjector-559cdd8c87-926k2\" (UID: \"0815c525-5349-44d2-a535-3db6bbe9521a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2"
Sep 16 18:45:54.125092 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:54.124962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0815c525-5349-44d2-a535-3db6bbe9521a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-926k2\" (UID: \"0815c525-5349-44d2-a535-3db6bbe9521a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2"
Sep 16 18:45:54.134021 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:54.133992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0815c525-5349-44d2-a535-3db6bbe9521a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-926k2\" (UID: \"0815c525-5349-44d2-a535-3db6bbe9521a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2"
Sep 16 18:45:54.134169 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:54.134149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8n6b\" (UniqueName: \"kubernetes.io/projected/0815c525-5349-44d2-a535-3db6bbe9521a-kube-api-access-s8n6b\") pod \"cert-manager-cainjector-559cdd8c87-926k2\" (UID: \"0815c525-5349-44d2-a535-3db6bbe9521a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2"
Sep 16 18:45:54.271211 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:54.271174    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2"
Sep 16 18:45:54.546181 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:54.546153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-926k2"]
Sep 16 18:45:54.547936 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:45:54.547907    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0815c525_5349_44d2_a535_3db6bbe9521a.slice/crio-373f080c2eba0dd6c5684d6ebf540268bb354964498c1f7f3665b0ad95bb4bc5 WatchSource:0}: Error finding container 373f080c2eba0dd6c5684d6ebf540268bb354964498c1f7f3665b0ad95bb4bc5: Status 404 returned error can't find the container with id 373f080c2eba0dd6c5684d6ebf540268bb354964498c1f7f3665b0ad95bb4bc5
Sep 16 18:45:55.424155 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:55.424116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2" event={"ID":"0815c525-5349-44d2-a535-3db6bbe9521a","Type":"ContainerStarted","Data":"4781b2f4544dfec26d65d824eee1fe9b94a862f0edbd6ecaaff2e0c41ac07e04"}
Sep 16 18:45:55.424679 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:55.424163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2" event={"ID":"0815c525-5349-44d2-a535-3db6bbe9521a","Type":"ContainerStarted","Data":"373f080c2eba0dd6c5684d6ebf540268bb354964498c1f7f3665b0ad95bb4bc5"}
Sep 16 18:45:58.417816 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:58.417782    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pt7kh"
Sep 16 18:45:58.436163 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:45:58.436108    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-926k2" podStartSLOduration=5.436090745 podStartE2EDuration="5.436090745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:55.452374569 +0000 UTC m=+397.240702935" watchObservedRunningTime="2026-09-16 18:45:58.436090745 +0000 UTC m=+400.224419110"
Sep 16 18:46:06.343784 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.343751    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jb8jw"]
Sep 16 18:46:06.347067 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.347048    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jb8jw"
Sep 16 18:46:06.349592 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.349558    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5whkt\""
Sep 16 18:46:06.363004 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.362980    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jb8jw"]
Sep 16 18:46:06.425181 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.425144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5a2cc49-05ff-4d83-986a-4f0d6247d9d1-bound-sa-token\") pod \"cert-manager-55b4d954d5-jb8jw\" (UID: \"f5a2cc49-05ff-4d83-986a-4f0d6247d9d1\") " pod="cert-manager/cert-manager-55b4d954d5-jb8jw"
Sep 16 18:46:06.425365 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.425193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kcbg\" (UniqueName: \"kubernetes.io/projected/f5a2cc49-05ff-4d83-986a-4f0d6247d9d1-kube-api-access-4kcbg\") pod \"cert-manager-55b4d954d5-jb8jw\" (UID: \"f5a2cc49-05ff-4d83-986a-4f0d6247d9d1\") " pod="cert-manager/cert-manager-55b4d954d5-jb8jw"
Sep 16 18:46:06.526615 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.526563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5a2cc49-05ff-4d83-986a-4f0d6247d9d1-bound-sa-token\") pod \"cert-manager-55b4d954d5-jb8jw\" (UID: \"f5a2cc49-05ff-4d83-986a-4f0d6247d9d1\") " pod="cert-manager/cert-manager-55b4d954d5-jb8jw"
Sep 16 18:46:06.526792 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.526621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcbg\" (UniqueName: \"kubernetes.io/projected/f5a2cc49-05ff-4d83-986a-4f0d6247d9d1-kube-api-access-4kcbg\") pod \"cert-manager-55b4d954d5-jb8jw\" (UID: \"f5a2cc49-05ff-4d83-986a-4f0d6247d9d1\") " pod="cert-manager/cert-manager-55b4d954d5-jb8jw"
Sep 16 18:46:06.535107 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.535080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5a2cc49-05ff-4d83-986a-4f0d6247d9d1-bound-sa-token\") pod \"cert-manager-55b4d954d5-jb8jw\" (UID: \"f5a2cc49-05ff-4d83-986a-4f0d6247d9d1\") " pod="cert-manager/cert-manager-55b4d954d5-jb8jw"
Sep 16 18:46:06.535237 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.535218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kcbg\" (UniqueName: \"kubernetes.io/projected/f5a2cc49-05ff-4d83-986a-4f0d6247d9d1-kube-api-access-4kcbg\") pod \"cert-manager-55b4d954d5-jb8jw\" (UID: \"f5a2cc49-05ff-4d83-986a-4f0d6247d9d1\") " pod="cert-manager/cert-manager-55b4d954d5-jb8jw"
Sep 16 18:46:06.656400 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.656304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jb8jw"
Sep 16 18:46:06.929549 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:06.929518    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jb8jw"]
Sep 16 18:46:06.932159 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:46:06.932132    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5a2cc49_05ff_4d83_986a_4f0d6247d9d1.slice/crio-2077ec3c5e07259da85c35dd3c4a75cc7b0b42fb49df38f0b899140c1ea5f212 WatchSource:0}: Error finding container 2077ec3c5e07259da85c35dd3c4a75cc7b0b42fb49df38f0b899140c1ea5f212: Status 404 returned error can't find the container with id 2077ec3c5e07259da85c35dd3c4a75cc7b0b42fb49df38f0b899140c1ea5f212
Sep 16 18:46:07.473070 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:07.473032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jb8jw" event={"ID":"f5a2cc49-05ff-4d83-986a-4f0d6247d9d1","Type":"ContainerStarted","Data":"87fbeb6eff5afcc6d20833fe85433bc2c1329802208e8cdc80a5d43e95a1b3f9"}
Sep 16 18:46:07.473070 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:07.473078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jb8jw" event={"ID":"f5a2cc49-05ff-4d83-986a-4f0d6247d9d1","Type":"ContainerStarted","Data":"2077ec3c5e07259da85c35dd3c4a75cc7b0b42fb49df38f0b899140c1ea5f212"}
Sep 16 18:46:07.494079 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:07.494025    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jb8jw" podStartSLOduration=1.494006915 podStartE2EDuration="1.494006915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:07.492933734 +0000 UTC m=+409.281262115" watchObservedRunningTime="2026-09-16 18:46:07.494006915 +0000 UTC m=+409.282335281"
Sep 16 18:46:30.787437 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.787397    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"]
Sep 16 18:46:30.794054 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.794030    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:30.797417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.797389    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:46:30.797723 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.797699    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8fq9q\""
Sep 16 18:46:30.797794 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.797712    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:46:30.797794 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.797775    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:46:30.797906 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.797800    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:30.797906 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.797889    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:30.805008 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.804983    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"]
Sep 16 18:46:30.944257 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.944210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3d8f65db-a55e-4803-9d99-b59827b5a570-manager-config\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:30.944439 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.944274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d8f65db-a55e-4803-9d99-b59827b5a570-metrics-cert\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:30.944439 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.944329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d8f65db-a55e-4803-9d99-b59827b5a570-cert\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:30.944439 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:30.944395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv8v2\" (UniqueName: \"kubernetes.io/projected/3d8f65db-a55e-4803-9d99-b59827b5a570-kube-api-access-gv8v2\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.045003 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.044904    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3d8f65db-a55e-4803-9d99-b59827b5a570-manager-config\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.045003 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.044962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d8f65db-a55e-4803-9d99-b59827b5a570-metrics-cert\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.045003 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.044987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d8f65db-a55e-4803-9d99-b59827b5a570-cert\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.045220 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.045015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8v2\" (UniqueName: \"kubernetes.io/projected/3d8f65db-a55e-4803-9d99-b59827b5a570-kube-api-access-gv8v2\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.045695 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.045667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3d8f65db-a55e-4803-9d99-b59827b5a570-manager-config\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.047447 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.047419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d8f65db-a55e-4803-9d99-b59827b5a570-metrics-cert\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.047708 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.047686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d8f65db-a55e-4803-9d99-b59827b5a570-cert\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.054581 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.054548    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8v2\" (UniqueName: \"kubernetes.io/projected/3d8f65db-a55e-4803-9d99-b59827b5a570-kube-api-access-gv8v2\") pod \"lws-controller-manager-856c96b694-z59vr\" (UID: \"3d8f65db-a55e-4803-9d99-b59827b5a570\") " pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.103822 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.103783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:31.385864 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.385725    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"]
Sep 16 18:46:31.388530 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:46:31.388498    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8f65db_a55e_4803_9d99_b59827b5a570.slice/crio-a1aa5c6c5513336bc1e28602f71f04577e2350769d789df3e3df541197c6edc2 WatchSource:0}: Error finding container a1aa5c6c5513336bc1e28602f71f04577e2350769d789df3e3df541197c6edc2: Status 404 returned error can't find the container with id a1aa5c6c5513336bc1e28602f71f04577e2350769d789df3e3df541197c6edc2
Sep 16 18:46:31.568414 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:31.568377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr" event={"ID":"3d8f65db-a55e-4803-9d99-b59827b5a570","Type":"ContainerStarted","Data":"a1aa5c6c5513336bc1e28602f71f04577e2350769d789df3e3df541197c6edc2"}
Sep 16 18:46:34.585071 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:34.585027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr" event={"ID":"3d8f65db-a55e-4803-9d99-b59827b5a570","Type":"ContainerStarted","Data":"20d7aaffb16d73fd59060bcc07979fccd00fb661b37f24ec2227e1ad058bb1ea"}
Sep 16 18:46:34.585447 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:34.585124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:34.611871 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:34.611815    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr" podStartSLOduration=1.89518461 podStartE2EDuration="4.61179772s" totalImagesPullingTime="2.71661311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:34.609959254 +0000 UTC m=+436.398287621" watchObservedRunningTime="2026-09-16 18:46:34.61179772 +0000 UTC m=+436.400126085"
Sep 16 18:46:45.593100 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:45.593022    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-856c96b694-z59vr"
Sep 16 18:46:53.635981 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.635950    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"]
Sep 16 18:46:53.639210 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.639191    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"
Sep 16 18:46:53.643146 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.643121    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-h2mgm\""
Sep 16 18:46:53.643296 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.643232    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:46:53.643352 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.643338    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:46:53.661072 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.661035    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"]
Sep 16 18:46:53.750478 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.750432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdt57\" (UniqueName: \"kubernetes.io/projected/659946bd-e2e2-4aed-95f0-5aeaeab3e745-kube-api-access-bdt57\") pod \"authorino-operator-6fc8f864f4-mwk6z\" (UID: \"659946bd-e2e2-4aed-95f0-5aeaeab3e745\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"
Sep 16 18:46:53.851637 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.851599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdt57\" (UniqueName: \"kubernetes.io/projected/659946bd-e2e2-4aed-95f0-5aeaeab3e745-kube-api-access-bdt57\") pod \"authorino-operator-6fc8f864f4-mwk6z\" (UID: \"659946bd-e2e2-4aed-95f0-5aeaeab3e745\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"
Sep 16 18:46:53.860495 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.860465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdt57\" (UniqueName: \"kubernetes.io/projected/659946bd-e2e2-4aed-95f0-5aeaeab3e745-kube-api-access-bdt57\") pod \"authorino-operator-6fc8f864f4-mwk6z\" (UID: \"659946bd-e2e2-4aed-95f0-5aeaeab3e745\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"
Sep 16 18:46:53.949620 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:53.949492    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"
Sep 16 18:46:54.231417 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:54.231389    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"]
Sep 16 18:46:54.233498 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:46:54.233469    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659946bd_e2e2_4aed_95f0_5aeaeab3e745.slice/crio-c325a9c2a931e57c32835435f7748dc4c0e09fbe46af5df78ee0bb478d19976b WatchSource:0}: Error finding container c325a9c2a931e57c32835435f7748dc4c0e09fbe46af5df78ee0bb478d19976b: Status 404 returned error can't find the container with id c325a9c2a931e57c32835435f7748dc4c0e09fbe46af5df78ee0bb478d19976b
Sep 16 18:46:54.669968 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:54.669883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z" event={"ID":"659946bd-e2e2-4aed-95f0-5aeaeab3e745","Type":"ContainerStarted","Data":"c325a9c2a931e57c32835435f7748dc4c0e09fbe46af5df78ee0bb478d19976b"}
Sep 16 18:46:57.687317 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:57.687279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z" event={"ID":"659946bd-e2e2-4aed-95f0-5aeaeab3e745","Type":"ContainerStarted","Data":"3fa573d1d0177bb7694e9541afb64bf4c33127319778e430f220e06f6dced155"}
Sep 16 18:46:57.687695 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:57.687422    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"
Sep 16 18:46:57.713046 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:46:57.712990    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z" podStartSLOduration=2.099566735 podStartE2EDuration="4.712972495s" totalImagesPullingTime="2.61340576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:57.711909526 +0000 UTC m=+459.500237915" watchObservedRunningTime="2026-09-16 18:46:57.712972495 +0000 UTC m=+459.501300861"
Sep 16 18:47:08.693696 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:08.693664    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-mwk6z"
Sep 16 18:47:37.080138 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.080099    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"]
Sep 16 18:47:37.088147 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.088125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:47:37.091051 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.091021    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:47:37.091702 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.091679    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-48428\""
Sep 16 18:47:37.092978 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.092955    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"]
Sep 16 18:47:37.143012 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.142976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99gw\" (UniqueName: \"kubernetes.io/projected/735d7064-203c-4a76-acba-2912d2ab58bf-kube-api-access-d99gw\") pod \"limitador-limitador-84994f7ffc-f9pfb\" (UID: \"735d7064-203c-4a76-acba-2912d2ab58bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:47:37.143209 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.143034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/735d7064-203c-4a76-acba-2912d2ab58bf-config-file\") pod \"limitador-limitador-84994f7ffc-f9pfb\" (UID: \"735d7064-203c-4a76-acba-2912d2ab58bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:47:37.144810 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.144786    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"]
Sep 16 18:47:37.244026 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.243980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99gw\" (UniqueName: \"kubernetes.io/projected/735d7064-203c-4a76-acba-2912d2ab58bf-kube-api-access-d99gw\") pod \"limitador-limitador-84994f7ffc-f9pfb\" (UID: \"735d7064-203c-4a76-acba-2912d2ab58bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:47:37.244216 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.244039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/735d7064-203c-4a76-acba-2912d2ab58bf-config-file\") pod \"limitador-limitador-84994f7ffc-f9pfb\" (UID: \"735d7064-203c-4a76-acba-2912d2ab58bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:47:37.244732 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.244711    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/735d7064-203c-4a76-acba-2912d2ab58bf-config-file\") pod \"limitador-limitador-84994f7ffc-f9pfb\" (UID: \"735d7064-203c-4a76-acba-2912d2ab58bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:47:37.256690 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.256651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99gw\" (UniqueName: \"kubernetes.io/projected/735d7064-203c-4a76-acba-2912d2ab58bf-kube-api-access-d99gw\") pod \"limitador-limitador-84994f7ffc-f9pfb\" (UID: \"735d7064-203c-4a76-acba-2912d2ab58bf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:47:37.399744 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.399647    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:47:37.683975 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.683942    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"]
Sep 16 18:47:37.686772 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:47:37.686745    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735d7064_203c_4a76_acba_2912d2ab58bf.slice/crio-78f3362725680a10c60b36a38c1c9880bcffff02fe26467b89069e9e754c6d73 WatchSource:0}: Error finding container 78f3362725680a10c60b36a38c1c9880bcffff02fe26467b89069e9e754c6d73: Status 404 returned error can't find the container with id 78f3362725680a10c60b36a38c1c9880bcffff02fe26467b89069e9e754c6d73
Sep 16 18:47:37.849696 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:37.849663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb" event={"ID":"735d7064-203c-4a76-acba-2912d2ab58bf","Type":"ContainerStarted","Data":"78f3362725680a10c60b36a38c1c9880bcffff02fe26467b89069e9e754c6d73"}
Sep 16 18:47:38.856529 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:38.856480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb" event={"ID":"735d7064-203c-4a76-acba-2912d2ab58bf","Type":"ContainerStarted","Data":"f92846fdd5666e620ccff6389e76a2106722aad46b66b89206a8e25df19990a2"}
Sep 16 18:47:38.856940 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:38.856614    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:47:38.875152 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:38.875095    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb" podStartSLOduration=0.917065635 podStartE2EDuration="1.875077545s" totalImagesPullingTime="958.01191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:38.874625572 +0000 UTC m=+500.662953939" watchObservedRunningTime="2026-09-16 18:47:38.875077545 +0000 UTC m=+500.663405901"
Sep 16 18:47:49.863313 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:47:49.863279    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-f9pfb"
Sep 16 18:49:18.668365 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:49:18.668329    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:49:18.672563 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:49:18.672533    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:52:18.080946 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.080914    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"]
Sep 16 18:52:18.084438 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.084415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.091301 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.090749    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:52:18.091938 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.091914    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:52:18.092407 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.092382    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-2grt6\""
Sep 16 18:52:18.096159 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.096136    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:52:18.101971 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.101942    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"]
Sep 16 18:52:18.203075 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.203044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.203245 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.203096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.203245 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.203149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brd4\" (UniqueName: \"kubernetes.io/projected/880388b6-7033-4ccd-b431-1835bf7e5642-kube-api-access-2brd4\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.203245 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.203168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-home\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.203245 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.203228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-dshm\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.203435 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.203247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880388b6-7033-4ccd-b431-1835bf7e5642-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.304222 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.304167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-dshm\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.304222 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.304224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880388b6-7033-4ccd-b431-1835bf7e5642-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.304445 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.304258    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.304445 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.304289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.304445 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.304311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brd4\" (UniqueName: \"kubernetes.io/projected/880388b6-7033-4ccd-b431-1835bf7e5642-kube-api-access-2brd4\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.304445 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.304330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-home\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.304713 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.304684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-home\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.304788 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.304735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.304845 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.304793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.306581 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.306549    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-dshm\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.306903 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.306887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880388b6-7033-4ccd-b431-1835bf7e5642-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.313041 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.313005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brd4\" (UniqueName: \"kubernetes.io/projected/880388b6-7033-4ccd-b431-1835bf7e5642-kube-api-access-2brd4\") pod \"fc-smoke-test-kserve-7f4687897-2c6xt\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.401229 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.401138    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:18.689066 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.688963    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"]
Sep 16 18:52:18.691286 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:52:18.691259    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880388b6_7033_4ccd_b431_1835bf7e5642.slice/crio-e5641743e1ac12df2e02c363244b08080d50d1e054897a5d6f4f1ad17db86175 WatchSource:0}: Error finding container e5641743e1ac12df2e02c363244b08080d50d1e054897a5d6f4f1ad17db86175: Status 404 returned error can't find the container with id e5641743e1ac12df2e02c363244b08080d50d1e054897a5d6f4f1ad17db86175
Sep 16 18:52:18.693241 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:18.693221    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:52:19.109659 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:19.109611    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt" event={"ID":"880388b6-7033-4ccd-b431-1835bf7e5642","Type":"ContainerStarted","Data":"e5641743e1ac12df2e02c363244b08080d50d1e054897a5d6f4f1ad17db86175"}
Sep 16 18:52:21.120431 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:21.120394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt" event={"ID":"880388b6-7033-4ccd-b431-1835bf7e5642","Type":"ContainerStarted","Data":"c766416c237b35104ba0b05925ab354ae18a57cc313a8e283388b031cb6916d0"}
Sep 16 18:52:21.152605 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:21.147101    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt" podStartSLOduration=1.618269482 podStartE2EDuration="3.147079181s" totalImagesPullingTime="1.528809699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:21.146206739 +0000 UTC m=+782.934535106" watchObservedRunningTime="2026-09-16 18:52:21.147079181 +0000 UTC m=+782.935407548"
Sep 16 18:52:21.402180 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:21.402082    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:28.401854 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:28.401813    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:28.414474 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:28.414444    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:29.168245 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:29.168210    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:52:59.970213 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:59.970169    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"]
Sep 16 18:52:59.981059 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:52:59.981009    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:52:59.981059 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:52:59.981059 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 18:52:59.981059 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="c766416c237b35104ba0b05925ab354ae18a57cc313a8e283388b031cb6916d0" cmd=["/bin/sleep","15"]
Sep 16 18:52:59.981404 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:59.981084    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt" message="" err=<
Sep 16 18:52:59.981404 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:52:59.981404 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 18:52:59.981404 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 18:52:59.981404 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:52:59.981098    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:52:59.981404 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:52:59.981404 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 18:52:59.981404 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt" podUID="880388b6-7033-4ccd-b431-1835bf7e5642" containerName="main" containerID="cri-o://c766416c237b35104ba0b05925ab354ae18a57cc313a8e283388b031cb6916d0"
Sep 16 18:52:59.981404 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:52:59.981158    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt" podUID="880388b6-7033-4ccd-b431-1835bf7e5642" containerName="main" containerID="cri-o://c766416c237b35104ba0b05925ab354ae18a57cc313a8e283388b031cb6916d0" gracePeriod=60
Sep 16 18:53:00.157728 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.157674    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt" podUID="880388b6-7033-4ccd-b431-1835bf7e5642" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 16 18:53:00.293841 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.293808    2561 generic.go:356] "Generic (PLEG): container finished" podID="880388b6-7033-4ccd-b431-1835bf7e5642" containerID="c766416c237b35104ba0b05925ab354ae18a57cc313a8e283388b031cb6916d0" exitCode=0
Sep 16 18:53:00.294004 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.293870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt" event={"ID":"880388b6-7033-4ccd-b431-1835bf7e5642","Type":"ContainerDied","Data":"c766416c237b35104ba0b05925ab354ae18a57cc313a8e283388b031cb6916d0"}
Sep 16 18:53:00.327638 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.327608    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:53:00.376211 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.376173    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-model-cache\") pod \"880388b6-7033-4ccd-b431-1835bf7e5642\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") "
Sep 16 18:53:00.376211 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.376205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-home\") pod \"880388b6-7033-4ccd-b431-1835bf7e5642\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") "
Sep 16 18:53:00.376471 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.376243    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-dshm\") pod \"880388b6-7033-4ccd-b431-1835bf7e5642\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") "
Sep 16 18:53:00.376471 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.376265    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880388b6-7033-4ccd-b431-1835bf7e5642-tls-certs\") pod \"880388b6-7033-4ccd-b431-1835bf7e5642\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") "
Sep 16 18:53:00.376471 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.376297    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-tmp-dir\") pod \"880388b6-7033-4ccd-b431-1835bf7e5642\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") "
Sep 16 18:53:00.376471 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.376349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2brd4\" (UniqueName: \"kubernetes.io/projected/880388b6-7033-4ccd-b431-1835bf7e5642-kube-api-access-2brd4\") pod \"880388b6-7033-4ccd-b431-1835bf7e5642\" (UID: \"880388b6-7033-4ccd-b431-1835bf7e5642\") "
Sep 16 18:53:00.376741 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.376515    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-model-cache" (OuterVolumeSpecName: "model-cache") pod "880388b6-7033-4ccd-b431-1835bf7e5642" (UID: "880388b6-7033-4ccd-b431-1835bf7e5642"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:00.376741 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.376544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-home" (OuterVolumeSpecName: "home") pod "880388b6-7033-4ccd-b431-1835bf7e5642" (UID: "880388b6-7033-4ccd-b431-1835bf7e5642"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:00.376741 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.376610    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "880388b6-7033-4ccd-b431-1835bf7e5642" (UID: "880388b6-7033-4ccd-b431-1835bf7e5642"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:00.378596 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.378530    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-dshm" (OuterVolumeSpecName: "dshm") pod "880388b6-7033-4ccd-b431-1835bf7e5642" (UID: "880388b6-7033-4ccd-b431-1835bf7e5642"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:00.378596 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.378545    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880388b6-7033-4ccd-b431-1835bf7e5642-kube-api-access-2brd4" (OuterVolumeSpecName: "kube-api-access-2brd4") pod "880388b6-7033-4ccd-b431-1835bf7e5642" (UID: "880388b6-7033-4ccd-b431-1835bf7e5642"). InnerVolumeSpecName "kube-api-access-2brd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:00.378596 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.378584    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880388b6-7033-4ccd-b431-1835bf7e5642-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "880388b6-7033-4ccd-b431-1835bf7e5642" (UID: "880388b6-7033-4ccd-b431-1835bf7e5642"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:53:00.477441 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.477399    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2brd4\" (UniqueName: \"kubernetes.io/projected/880388b6-7033-4ccd-b431-1835bf7e5642-kube-api-access-2brd4\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:53:00.477441 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.477432    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:53:00.477441 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.477442    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:53:00.477441 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.477450    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:53:00.477737 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.477459    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880388b6-7033-4ccd-b431-1835bf7e5642-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:53:00.477737 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:00.477466    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880388b6-7033-4ccd-b431-1835bf7e5642-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:53:01.299877 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:01.299837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt" event={"ID":"880388b6-7033-4ccd-b431-1835bf7e5642","Type":"ContainerDied","Data":"e5641743e1ac12df2e02c363244b08080d50d1e054897a5d6f4f1ad17db86175"}
Sep 16 18:53:01.299877 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:01.299869    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"
Sep 16 18:53:01.300389 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:01.299901    2561 scope.go:117] "RemoveContainer" containerID="c766416c237b35104ba0b05925ab354ae18a57cc313a8e283388b031cb6916d0"
Sep 16 18:53:01.333982 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:01.333939    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"]
Sep 16 18:53:01.339583 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:01.339533    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-2c6xt"]
Sep 16 18:53:02.749373 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:02.749333    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="880388b6-7033-4ccd-b431-1835bf7e5642" path="/var/lib/kubelet/pods/880388b6-7033-4ccd-b431-1835bf7e5642/volumes"
Sep 16 18:53:56.107150 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.107107    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"]
Sep 16 18:53:56.107655 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.107465    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="880388b6-7033-4ccd-b431-1835bf7e5642" containerName="main"
Sep 16 18:53:56.107655 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.107478    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="880388b6-7033-4ccd-b431-1835bf7e5642" containerName="main"
Sep 16 18:53:56.107655 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.107529    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="880388b6-7033-4ccd-b431-1835bf7e5642" containerName="main"
Sep 16 18:53:56.110608 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.110591    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.114178 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.114156    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 18:53:56.114313 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.114214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 18:53:56.114313 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.114235    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-2swtq\""
Sep 16 18:53:56.114313 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.114296    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 18:53:56.124924 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.124895    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"]
Sep 16 18:53:56.243797 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.243756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.243797 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.243799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-home\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.244019 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.243824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.244019 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.243903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82d43ed-a421-4760-863c-d417bc4ef1cd-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.244019 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.243933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsqc7\" (UniqueName: \"kubernetes.io/projected/b82d43ed-a421-4760-863c-d417bc4ef1cd-kube-api-access-dsqc7\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.244019 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.243965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.345373 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.345333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.345373 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.345375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-home\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.345822 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.345408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.345822 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.345464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82d43ed-a421-4760-863c-d417bc4ef1cd-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.345822 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.345488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsqc7\" (UniqueName: \"kubernetes.io/projected/b82d43ed-a421-4760-863c-d417bc4ef1cd-kube-api-access-dsqc7\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.345822 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.345526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.346148 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.345823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.346148 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.345864    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-home\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.346148 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.346097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.348001 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.347971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.348533 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.348510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82d43ed-a421-4760-863c-d417bc4ef1cd-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.359154 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.359079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsqc7\" (UniqueName: \"kubernetes.io/projected/b82d43ed-a421-4760-863c-d417bc4ef1cd-kube-api-access-dsqc7\") pod \"fc-concurrency-test-kserve-b5d44467c-vpxsk\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.420753 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.420702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:53:56.697010 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:56.696985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"]
Sep 16 18:53:56.698827 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:53:56.698799    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82d43ed_a421_4760_863c_d417bc4ef1cd.slice/crio-5c83b1c0cbc26ebddd685c455a16fcb9df11d38322879b972fb067d9d4fa6b57 WatchSource:0}: Error finding container 5c83b1c0cbc26ebddd685c455a16fcb9df11d38322879b972fb067d9d4fa6b57: Status 404 returned error can't find the container with id 5c83b1c0cbc26ebddd685c455a16fcb9df11d38322879b972fb067d9d4fa6b57
Sep 16 18:53:57.554159 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:57.554120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk" event={"ID":"b82d43ed-a421-4760-863c-d417bc4ef1cd","Type":"ContainerStarted","Data":"a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc"}
Sep 16 18:53:57.554560 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:57.554176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk" event={"ID":"b82d43ed-a421-4760-863c-d417bc4ef1cd","Type":"ContainerStarted","Data":"5c83b1c0cbc26ebddd685c455a16fcb9df11d38322879b972fb067d9d4fa6b57"}
Sep 16 18:53:57.583490 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:57.583432    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk" podStartSLOduration=1.583412809 podStartE2EDuration="1.583412809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:57.583089421 +0000 UTC m=+879.371417788" watchObservedRunningTime="2026-09-16 18:53:57.583412809 +0000 UTC m=+879.371741176"
Sep 16 18:53:58.421087 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:53:58.421055    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:54:06.421018 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:06.420975    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:54:06.433662 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:06.433632    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:54:06.606466 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:06.606435    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:54:18.698192 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:18.698115    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:54:18.698746 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:18.698299    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:54:35.194660 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.194623    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"]
Sep 16 18:54:35.205426 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:54:35.205385    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:54:35.205426 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:54:35.205426 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 18:54:35.205426 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc" cmd=["/bin/sleep","15"]
Sep 16 18:54:35.205777 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.205442    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk" message="" err=<
Sep 16 18:54:35.205777 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:54:35.205777 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 18:54:35.205777 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 18:54:35.205777 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:54:35.205454    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:54:35.205777 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:54:35.205777 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 18:54:35.205777 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk" podUID="b82d43ed-a421-4760-863c-d417bc4ef1cd" containerName="main" containerID="cri-o://a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc"
Sep 16 18:54:35.205777 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.205499    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk" podUID="b82d43ed-a421-4760-863c-d417bc4ef1cd" containerName="main" containerID="cri-o://a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc" gracePeriod=60
Sep 16 18:54:35.560923 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.560898    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:54:35.596587 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.596547    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-home\") pod \"b82d43ed-a421-4760-863c-d417bc4ef1cd\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") "
Sep 16 18:54:35.596771 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.596609    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-tmp-dir\") pod \"b82d43ed-a421-4760-863c-d417bc4ef1cd\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") "
Sep 16 18:54:35.596771 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.596625    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-dshm\") pod \"b82d43ed-a421-4760-863c-d417bc4ef1cd\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") "
Sep 16 18:54:35.596771 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.596677    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82d43ed-a421-4760-863c-d417bc4ef1cd-tls-certs\") pod \"b82d43ed-a421-4760-863c-d417bc4ef1cd\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") "
Sep 16 18:54:35.596771 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.596697    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsqc7\" (UniqueName: \"kubernetes.io/projected/b82d43ed-a421-4760-863c-d417bc4ef1cd-kube-api-access-dsqc7\") pod \"b82d43ed-a421-4760-863c-d417bc4ef1cd\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") "
Sep 16 18:54:35.596771 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.596756    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-model-cache\") pod \"b82d43ed-a421-4760-863c-d417bc4ef1cd\" (UID: \"b82d43ed-a421-4760-863c-d417bc4ef1cd\") "
Sep 16 18:54:35.597056 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.596848    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-home" (OuterVolumeSpecName: "home") pod "b82d43ed-a421-4760-863c-d417bc4ef1cd" (UID: "b82d43ed-a421-4760-863c-d417bc4ef1cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:35.597056 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.596865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b82d43ed-a421-4760-863c-d417bc4ef1cd" (UID: "b82d43ed-a421-4760-863c-d417bc4ef1cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:35.597056 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.597024    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:54:35.597056 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.597045    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:54:35.597266 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.597057    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b82d43ed-a421-4760-863c-d417bc4ef1cd" (UID: "b82d43ed-a421-4760-863c-d417bc4ef1cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:35.599030 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.599002    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-dshm" (OuterVolumeSpecName: "dshm") pod "b82d43ed-a421-4760-863c-d417bc4ef1cd" (UID: "b82d43ed-a421-4760-863c-d417bc4ef1cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:35.599030 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.599011    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82d43ed-a421-4760-863c-d417bc4ef1cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b82d43ed-a421-4760-863c-d417bc4ef1cd" (UID: "b82d43ed-a421-4760-863c-d417bc4ef1cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:35.599188 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.599038    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82d43ed-a421-4760-863c-d417bc4ef1cd-kube-api-access-dsqc7" (OuterVolumeSpecName: "kube-api-access-dsqc7") pod "b82d43ed-a421-4760-863c-d417bc4ef1cd" (UID: "b82d43ed-a421-4760-863c-d417bc4ef1cd"). InnerVolumeSpecName "kube-api-access-dsqc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:35.698200 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.698172    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:54:35.698200 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.698197    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82d43ed-a421-4760-863c-d417bc4ef1cd-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:54:35.698397 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.698213    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsqc7\" (UniqueName: \"kubernetes.io/projected/b82d43ed-a421-4760-863c-d417bc4ef1cd-kube-api-access-dsqc7\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:54:35.698397 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.698225    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82d43ed-a421-4760-863c-d417bc4ef1cd-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:54:35.728487 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.728452    2561 generic.go:356] "Generic (PLEG): container finished" podID="b82d43ed-a421-4760-863c-d417bc4ef1cd" containerID="a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc" exitCode=0
Sep 16 18:54:35.728663 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.728497    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"
Sep 16 18:54:35.728663 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.728544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk" event={"ID":"b82d43ed-a421-4760-863c-d417bc4ef1cd","Type":"ContainerDied","Data":"a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc"}
Sep 16 18:54:35.728663 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.728630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk" event={"ID":"b82d43ed-a421-4760-863c-d417bc4ef1cd","Type":"ContainerDied","Data":"5c83b1c0cbc26ebddd685c455a16fcb9df11d38322879b972fb067d9d4fa6b57"}
Sep 16 18:54:35.728663 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.728653    2561 scope.go:117] "RemoveContainer" containerID="a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc"
Sep 16 18:54:35.737818 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.737792    2561 scope.go:117] "RemoveContainer" containerID="a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc"
Sep 16 18:54:35.738101 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:54:35.738081    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc\": container with ID starting with a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc not found: ID does not exist" containerID="a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc"
Sep 16 18:54:35.738170 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.738108    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc"} err="failed to get container status \"a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc\": rpc error: code = NotFound desc = could not find container \"a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc\": container with ID starting with a38d87902ccf33fa298f67bd7eca96b92c4f969c2338ae002910704f8c2a35cc not found: ID does not exist"
Sep 16 18:54:35.751864 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.751834    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"]
Sep 16 18:54:35.755972 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:35.755934    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpxsk"]
Sep 16 18:54:36.750225 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:54:36.750192    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b82d43ed-a421-4760-863c-d417bc4ef1cd" path="/var/lib/kubelet/pods/b82d43ed-a421-4760-863c-d417bc4ef1cd/volumes"
Sep 16 18:55:05.302107 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.302072    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:55:05.302543 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.302407    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b82d43ed-a421-4760-863c-d417bc4ef1cd" containerName="main"
Sep 16 18:55:05.302543 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.302418    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b82d43ed-a421-4760-863c-d417bc4ef1cd" containerName="main"
Sep 16 18:55:05.302543 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.302475    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b82d43ed-a421-4760-863c-d417bc4ef1cd" containerName="main"
Sep 16 18:55:05.305510 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.305487    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.309400 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.309367    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:55:05.310180 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.310155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-cx5b4\""
Sep 16 18:55:05.310380 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.310359    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:55:05.310441 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.310428    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:55:05.310494 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.310456    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:55:05.310962 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.310943    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:55:05.311638 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.311450    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-w98nl\""
Sep 16 18:55:05.326099 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.325991    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:55:05.373755 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.373707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.373963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.373770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.373963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.373822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.373963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.373849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.373963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.373874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f9cbf4-c101-4e9c-adda-15470eac1f71-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.373963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.373896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.373963 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.373914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f9cbf4-c101-4e9c-adda-15470eac1f71-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.374265 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.373963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6mt9\" (UniqueName: \"kubernetes.io/projected/64f9cbf4-c101-4e9c-adda-15470eac1f71-kube-api-access-m6mt9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.474854 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.474814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475030 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.474870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475030 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.474919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475030 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.474948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475030 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.474975    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f9cbf4-c101-4e9c-adda-15470eac1f71-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475237 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.475124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475237 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.475178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f9cbf4-c101-4e9c-adda-15470eac1f71-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475237 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.475214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6mt9\" (UniqueName: \"kubernetes.io/projected/64f9cbf4-c101-4e9c-adda-15470eac1f71-kube-api-access-m6mt9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475392 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.475320    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475474 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.475447    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475533 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.475458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475666 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.475645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.475724 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.475669    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f9cbf4-c101-4e9c-adda-15470eac1f71-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.477118 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.477091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.477588 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.477552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f9cbf4-c101-4e9c-adda-15470eac1f71-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.483442 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.483418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6mt9\" (UniqueName: \"kubernetes.io/projected/64f9cbf4-c101-4e9c-adda-15470eac1f71-kube-api-access-m6mt9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.627718 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.627622    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:05.923150 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:05.923111    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:55:05.924598 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:55:05.924543    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f9cbf4_c101_4e9c_adda_15470eac1f71.slice/crio-e572c4bfb171aaad209a92c054b2a2367a3a0329d5f24e92683dc5b57d4cc6ba WatchSource:0}: Error finding container e572c4bfb171aaad209a92c054b2a2367a3a0329d5f24e92683dc5b57d4cc6ba: Status 404 returned error can't find the container with id e572c4bfb171aaad209a92c054b2a2367a3a0329d5f24e92683dc5b57d4cc6ba
Sep 16 18:55:06.866077 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:06.866037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"64f9cbf4-c101-4e9c-adda-15470eac1f71","Type":"ContainerStarted","Data":"e572c4bfb171aaad209a92c054b2a2367a3a0329d5f24e92683dc5b57d4cc6ba"}
Sep 16 18:55:09.883770 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:09.883736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"64f9cbf4-c101-4e9c-adda-15470eac1f71","Type":"ContainerStarted","Data":"acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed"}
Sep 16 18:55:11.894100 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:11.894011    2561 generic.go:356] "Generic (PLEG): container finished" podID="64f9cbf4-c101-4e9c-adda-15470eac1f71" containerID="acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed" exitCode=0
Sep 16 18:55:11.894100 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:11.894083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"64f9cbf4-c101-4e9c-adda-15470eac1f71","Type":"ContainerDied","Data":"acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed"}
Sep 16 18:55:42.067699 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:42.067660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"64f9cbf4-c101-4e9c-adda-15470eac1f71","Type":"ContainerStarted","Data":"47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f"}
Sep 16 18:55:42.107126 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:55:42.107059    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.013645966 podStartE2EDuration="37.107036431s" totalImagesPullingTime="32.901045454s" totalInitContainerRuntime="2.192345011s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:42.100026966 +0000 UTC m=+983.888355333" watchObservedRunningTime="2026-09-16 18:55:42.107036431 +0000 UTC m=+983.895364800"
Sep 16 18:57:17.084058 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:17.083985    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:57:32.105402 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.105341    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="64f9cbf4-c101-4e9c-adda-15470eac1f71" containerName="main" containerID="cri-o://47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f" gracePeriod=45
Sep 16 18:57:32.856382 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.856352    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:32.936498 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936418    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f9cbf4-c101-4e9c-adda-15470eac1f71-cabundle-cert\") pod \"64f9cbf4-c101-4e9c-adda-15470eac1f71\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") "
Sep 16 18:57:32.936498 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-home\") pod \"64f9cbf4-c101-4e9c-adda-15470eac1f71\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") "
Sep 16 18:57:32.936791 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936600    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6mt9\" (UniqueName: \"kubernetes.io/projected/64f9cbf4-c101-4e9c-adda-15470eac1f71-kube-api-access-m6mt9\") pod \"64f9cbf4-c101-4e9c-adda-15470eac1f71\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") "
Sep 16 18:57:32.936791 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936652    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f9cbf4-c101-4e9c-adda-15470eac1f71-tls-certs\") pod \"64f9cbf4-c101-4e9c-adda-15470eac1f71\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") "
Sep 16 18:57:32.936791 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936673    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-model-cache\") pod \"64f9cbf4-c101-4e9c-adda-15470eac1f71\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") "
Sep 16 18:57:32.936791 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936689    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-tmp-dir\") pod \"64f9cbf4-c101-4e9c-adda-15470eac1f71\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") "
Sep 16 18:57:32.936791 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936732    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-kserve-provision-location\") pod \"64f9cbf4-c101-4e9c-adda-15470eac1f71\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") "
Sep 16 18:57:32.936791 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936762    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-dshm\") pod \"64f9cbf4-c101-4e9c-adda-15470eac1f71\" (UID: \"64f9cbf4-c101-4e9c-adda-15470eac1f71\") "
Sep 16 18:57:32.937095 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f9cbf4-c101-4e9c-adda-15470eac1f71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64f9cbf4-c101-4e9c-adda-15470eac1f71" (UID: "64f9cbf4-c101-4e9c-adda-15470eac1f71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:32.937095 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.936977    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-model-cache" (OuterVolumeSpecName: "model-cache") pod "64f9cbf4-c101-4e9c-adda-15470eac1f71" (UID: "64f9cbf4-c101-4e9c-adda-15470eac1f71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:32.937095 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.937071    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:57:32.937095 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.937091    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f9cbf4-c101-4e9c-adda-15470eac1f71-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:57:32.937304 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.937189    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-home" (OuterVolumeSpecName: "home") pod "64f9cbf4-c101-4e9c-adda-15470eac1f71" (UID: "64f9cbf4-c101-4e9c-adda-15470eac1f71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:32.938994 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.938970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-dshm" (OuterVolumeSpecName: "dshm") pod "64f9cbf4-c101-4e9c-adda-15470eac1f71" (UID: "64f9cbf4-c101-4e9c-adda-15470eac1f71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:32.938994 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.938980    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f9cbf4-c101-4e9c-adda-15470eac1f71-kube-api-access-m6mt9" (OuterVolumeSpecName: "kube-api-access-m6mt9") pod "64f9cbf4-c101-4e9c-adda-15470eac1f71" (UID: "64f9cbf4-c101-4e9c-adda-15470eac1f71"). InnerVolumeSpecName "kube-api-access-m6mt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:32.939168 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.939070    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f9cbf4-c101-4e9c-adda-15470eac1f71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64f9cbf4-c101-4e9c-adda-15470eac1f71" (UID: "64f9cbf4-c101-4e9c-adda-15470eac1f71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:32.949235 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.949197    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64f9cbf4-c101-4e9c-adda-15470eac1f71" (UID: "64f9cbf4-c101-4e9c-adda-15470eac1f71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:32.956934 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:32.956896    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64f9cbf4-c101-4e9c-adda-15470eac1f71" (UID: "64f9cbf4-c101-4e9c-adda-15470eac1f71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:33.038372 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.038335    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:57:33.038372 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.038367    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6mt9\" (UniqueName: \"kubernetes.io/projected/64f9cbf4-c101-4e9c-adda-15470eac1f71-kube-api-access-m6mt9\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:57:33.038372 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.038380    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f9cbf4-c101-4e9c-adda-15470eac1f71-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:57:33.038631 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.038409    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:57:33.038631 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.038418    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:57:33.038631 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.038426    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f9cbf4-c101-4e9c-adda-15470eac1f71-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:57:33.600331 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.600294    2561 generic.go:356] "Generic (PLEG): container finished" podID="64f9cbf4-c101-4e9c-adda-15470eac1f71" containerID="47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f" exitCode=0
Sep 16 18:57:33.600820 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.600352    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:57:33.600820 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.600352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"64f9cbf4-c101-4e9c-adda-15470eac1f71","Type":"ContainerDied","Data":"47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f"}
Sep 16 18:57:33.600820 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.600496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"64f9cbf4-c101-4e9c-adda-15470eac1f71","Type":"ContainerDied","Data":"e572c4bfb171aaad209a92c054b2a2367a3a0329d5f24e92683dc5b57d4cc6ba"}
Sep 16 18:57:33.600820 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.600538    2561 scope.go:117] "RemoveContainer" containerID="47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f"
Sep 16 18:57:33.609227 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.609208    2561 scope.go:117] "RemoveContainer" containerID="acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed"
Sep 16 18:57:33.616523 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.616505    2561 scope.go:117] "RemoveContainer" containerID="47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f"
Sep 16 18:57:33.616849 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:57:33.616832    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f\": container with ID starting with 47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f not found: ID does not exist" containerID="47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f"
Sep 16 18:57:33.616892 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.616858    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f"} err="failed to get container status \"47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f\": rpc error: code = NotFound desc = could not find container \"47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f\": container with ID starting with 47c6399ad2d1568792adee7ab267e550775fb282fb524bb49217a75ec4c0e95f not found: ID does not exist"
Sep 16 18:57:33.616892 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.616876    2561 scope.go:117] "RemoveContainer" containerID="acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed"
Sep 16 18:57:33.617103 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:57:33.617086    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed\": container with ID starting with acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed not found: ID does not exist" containerID="acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed"
Sep 16 18:57:33.617141 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.617111    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed"} err="failed to get container status \"acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed\": rpc error: code = NotFound desc = could not find container \"acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed\": container with ID starting with acab1d1566b7e1b40974bf9680295fd05de548b19aa5f9b5907d9ecfe93a08ed not found: ID does not exist"
Sep 16 18:57:33.629225 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.629204    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:57:33.635385 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:33.635364    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:57:34.748925 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:57:34.748888    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f9cbf4-c101-4e9c-adda-15470eac1f71" path="/var/lib/kubelet/pods/64f9cbf4-c101-4e9c-adda-15470eac1f71/volumes"
Sep 16 18:58:01.664241 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.664202    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"]
Sep 16 18:58:01.664662 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.664551    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f9cbf4-c101-4e9c-adda-15470eac1f71" containerName="main"
Sep 16 18:58:01.664662 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.664579    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f9cbf4-c101-4e9c-adda-15470eac1f71" containerName="main"
Sep 16 18:58:01.664662 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.664603    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f9cbf4-c101-4e9c-adda-15470eac1f71" containerName="storage-initializer"
Sep 16 18:58:01.664662 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.664609    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f9cbf4-c101-4e9c-adda-15470eac1f71" containerName="storage-initializer"
Sep 16 18:58:01.664662 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.664661    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f9cbf4-c101-4e9c-adda-15470eac1f71" containerName="main"
Sep 16 18:58:01.667609 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.667586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.670288 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.670263    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 18:58:01.670652 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.670632    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 18:58:01.670760 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:58:01.670644    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-pmxtc\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-47655ff5\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-pmxtc\"" type="*v1.Secret"
Sep 16 18:58:01.670760 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:58:01.670747    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" is forbidden: User \"system:node:ip-10-0-142-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-47655ff5\": no relationship found between node 'ip-10-0-142-197.ec2.internal' and this object" podUID="35b458ea-c875-49c6-8f2e-db37661888aa" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.671196 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.671178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 18:58:01.676940 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.676900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.676940 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.676942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.677119 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.676967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.677119 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.677009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6276\" (UniqueName: \"kubernetes.io/projected/35b458ea-c875-49c6-8f2e-db37661888aa-kube-api-access-s6276\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.677119 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.677059    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b458ea-c875-49c6-8f2e-db37661888aa-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.677119 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.677082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.680143 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.680116    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"]
Sep 16 18:58:01.777881 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.777847    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.777881 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.777882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.778119 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.777903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.778119 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.777930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6276\" (UniqueName: \"kubernetes.io/projected/35b458ea-c875-49c6-8f2e-db37661888aa-kube-api-access-s6276\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.778119 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.777996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b458ea-c875-49c6-8f2e-db37661888aa-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.778119 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.778032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.778339 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.778315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.778403 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.778372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.778403 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.778392    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.780266 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.780236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.780545 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.780523    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b458ea-c875-49c6-8f2e-db37661888aa-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:01.787281 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:01.787243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6276\" (UniqueName: \"kubernetes.io/projected/35b458ea-c875-49c6-8f2e-db37661888aa-kube-api-access-s6276\") pod \"scheduler-inline-config-test-kserve-8655bc757-9fffs\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:02.916292 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:02.916258    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-pmxtc\""
Sep 16 18:58:02.919326 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:02.919293    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:03.193404 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:03.193377    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"]
Sep 16 18:58:03.196284 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:58:03.196254    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b458ea_c875_49c6_8f2e_db37661888aa.slice/crio-6ab012e46c50a5b6ae2cb54629f099179e842f2db8ba85e21256aeb1da894977 WatchSource:0}: Error finding container 6ab012e46c50a5b6ae2cb54629f099179e842f2db8ba85e21256aeb1da894977: Status 404 returned error can't find the container with id 6ab012e46c50a5b6ae2cb54629f099179e842f2db8ba85e21256aeb1da894977
Sep 16 18:58:03.736847 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:03.736813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs" event={"ID":"35b458ea-c875-49c6-8f2e-db37661888aa","Type":"ContainerStarted","Data":"9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b"}
Sep 16 18:58:03.737039 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:03.736862    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs" event={"ID":"35b458ea-c875-49c6-8f2e-db37661888aa","Type":"ContainerStarted","Data":"6ab012e46c50a5b6ae2cb54629f099179e842f2db8ba85e21256aeb1da894977"}
Sep 16 18:58:03.757317 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:03.757258    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs" podStartSLOduration=2.757238297 podStartE2EDuration="2.757238297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:03.755429754 +0000 UTC m=+1125.543758119" watchObservedRunningTime="2026-09-16 18:58:03.757238297 +0000 UTC m=+1125.545566662"
Sep 16 18:58:03.919516 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:03.919472    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:12.919477 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:12.919427    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:12.932077 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:12.932049    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:13.789856 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:13.789825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:36.291680 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.291639    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"]
Sep 16 18:58:36.300775 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:58:36.300729    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:58:36.300775 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:58:36.300775 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 18:58:36.300775 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b" cmd=["/bin/sleep","15"]
Sep 16 18:58:36.301072 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.300797    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs" message="" err=<
Sep 16 18:58:36.301072 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:58:36.301072 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 18:58:36.301072 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 18:58:36.301072 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:58:36.300816    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:58:36.301072 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:58:36.301072 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 18:58:36.301072 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs" podUID="35b458ea-c875-49c6-8f2e-db37661888aa" containerName="main" containerID="cri-o://9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b"
Sep 16 18:58:36.301072 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.300876    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs" podUID="35b458ea-c875-49c6-8f2e-db37661888aa" containerName="main" containerID="cri-o://9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b" gracePeriod=60
Sep 16 18:58:36.654183 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.654155    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:36.785047 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.785000    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6276\" (UniqueName: \"kubernetes.io/projected/35b458ea-c875-49c6-8f2e-db37661888aa-kube-api-access-s6276\") pod \"35b458ea-c875-49c6-8f2e-db37661888aa\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") "
Sep 16 18:58:36.785239 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.785087    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-dshm\") pod \"35b458ea-c875-49c6-8f2e-db37661888aa\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") "
Sep 16 18:58:36.785239 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.785128    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-tmp-dir\") pod \"35b458ea-c875-49c6-8f2e-db37661888aa\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") "
Sep 16 18:58:36.785239 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.785151    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-home\") pod \"35b458ea-c875-49c6-8f2e-db37661888aa\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") "
Sep 16 18:58:36.785239 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.785171    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-model-cache\") pod \"35b458ea-c875-49c6-8f2e-db37661888aa\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") "
Sep 16 18:58:36.785239 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.785200    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b458ea-c875-49c6-8f2e-db37661888aa-tls-certs\") pod \"35b458ea-c875-49c6-8f2e-db37661888aa\" (UID: \"35b458ea-c875-49c6-8f2e-db37661888aa\") "
Sep 16 18:58:36.785507 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.785391    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-home" (OuterVolumeSpecName: "home") pod "35b458ea-c875-49c6-8f2e-db37661888aa" (UID: "35b458ea-c875-49c6-8f2e-db37661888aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:36.785507 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.785424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35b458ea-c875-49c6-8f2e-db37661888aa" (UID: "35b458ea-c875-49c6-8f2e-db37661888aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:36.785507 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.785440    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "35b458ea-c875-49c6-8f2e-db37661888aa" (UID: "35b458ea-c875-49c6-8f2e-db37661888aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:36.787351 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.787324    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-dshm" (OuterVolumeSpecName: "dshm") pod "35b458ea-c875-49c6-8f2e-db37661888aa" (UID: "35b458ea-c875-49c6-8f2e-db37661888aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:36.787436 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.787411    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b458ea-c875-49c6-8f2e-db37661888aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35b458ea-c875-49c6-8f2e-db37661888aa" (UID: "35b458ea-c875-49c6-8f2e-db37661888aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:36.787486 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.787470    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b458ea-c875-49c6-8f2e-db37661888aa-kube-api-access-s6276" (OuterVolumeSpecName: "kube-api-access-s6276") pod "35b458ea-c875-49c6-8f2e-db37661888aa" (UID: "35b458ea-c875-49c6-8f2e-db37661888aa"). InnerVolumeSpecName "kube-api-access-s6276". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:36.883371 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.883288    2561 generic.go:356] "Generic (PLEG): container finished" podID="35b458ea-c875-49c6-8f2e-db37661888aa" containerID="9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b" exitCode=0
Sep 16 18:58:36.883371 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.883341    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"
Sep 16 18:58:36.883371 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.883352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs" event={"ID":"35b458ea-c875-49c6-8f2e-db37661888aa","Type":"ContainerDied","Data":"9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b"}
Sep 16 18:58:36.883665 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.883396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs" event={"ID":"35b458ea-c875-49c6-8f2e-db37661888aa","Type":"ContainerDied","Data":"6ab012e46c50a5b6ae2cb54629f099179e842f2db8ba85e21256aeb1da894977"}
Sep 16 18:58:36.883665 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.883416    2561 scope.go:117] "RemoveContainer" containerID="9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b"
Sep 16 18:58:36.886119 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.886099    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6276\" (UniqueName: \"kubernetes.io/projected/35b458ea-c875-49c6-8f2e-db37661888aa-kube-api-access-s6276\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.886203 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.886122    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.886203 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.886155    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.886203 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.886166    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.886203 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.886173    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35b458ea-c875-49c6-8f2e-db37661888aa-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.886203 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.886181    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35b458ea-c875-49c6-8f2e-db37661888aa-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.892634 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.892617    2561 scope.go:117] "RemoveContainer" containerID="9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b"
Sep 16 18:58:36.892936 ip-10-0-142-197 kubenswrapper[2561]: E0916 18:58:36.892916    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b\": container with ID starting with 9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b not found: ID does not exist" containerID="9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b"
Sep 16 18:58:36.892997 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.892943    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b"} err="failed to get container status \"9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b\": rpc error: code = NotFound desc = could not find container \"9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b\": container with ID starting with 9155d8da58354586033b13d0524a243341df8e9702f42fc7d7a9c614ae97f65b not found: ID does not exist"
Sep 16 18:58:36.918548 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.918510    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"]
Sep 16 18:58:36.957829 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:36.957792    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-9fffs"]
Sep 16 18:58:38.750309 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:58:38.750270    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b458ea-c875-49c6-8f2e-db37661888aa" path="/var/lib/kubelet/pods/35b458ea-c875-49c6-8f2e-db37661888aa/volumes"
Sep 16 18:59:13.242292 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.242253    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"]
Sep 16 18:59:13.242772 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.242606    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b458ea-c875-49c6-8f2e-db37661888aa" containerName="main"
Sep 16 18:59:13.242772 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.242620    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b458ea-c875-49c6-8f2e-db37661888aa" containerName="main"
Sep 16 18:59:13.242772 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.242700    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b458ea-c875-49c6-8f2e-db37661888aa" containerName="main"
Sep 16 18:59:13.245784 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.245766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 18:59:13.250257 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.250237    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-6d5ln\""
Sep 16 18:59:13.250392 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.250240    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 18:59:13.250392 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.250293    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 18:59:13.250392 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.250239    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-s7w7l\""
Sep 16 18:59:13.251407 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.251388    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 18:59:13.259436 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.259414    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"]
Sep 16 18:59:13.275269 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.275231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmfn\" (UniqueName: \"kubernetes.io/projected/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-kube-api-access-qdmfn\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n\" (UID: \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 18:59:13.275412 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.275289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n\" (UID: \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 18:59:13.376435 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.376400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmfn\" (UniqueName: \"kubernetes.io/projected/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-kube-api-access-qdmfn\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n\" (UID: \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 18:59:13.376650 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.376460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n\" (UID: \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 18:59:13.378921 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.378891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n\" (UID: \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 18:59:13.388552 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.388520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmfn\" (UniqueName: \"kubernetes.io/projected/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-kube-api-access-qdmfn\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n\" (UID: \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 18:59:13.556231 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.556132    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 18:59:13.838067 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.838041    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"]
Sep 16 18:59:13.840337 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:59:13.840306    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ca5cbf_7961_4ba3_b39e_42e571bb266c.slice/crio-90711457780afa47cde18c326ca548c61642c1018dc2a6184cf11fc24d9c2e1f WatchSource:0}: Error finding container 90711457780afa47cde18c326ca548c61642c1018dc2a6184cf11fc24d9c2e1f: Status 404 returned error can't find the container with id 90711457780afa47cde18c326ca548c61642c1018dc2a6184cf11fc24d9c2e1f
Sep 16 18:59:13.842452 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:13.842433    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:59:14.047057 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:14.047025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" event={"ID":"b1ca5cbf-7961-4ba3-b39e-42e571bb266c","Type":"ContainerStarted","Data":"90711457780afa47cde18c326ca548c61642c1018dc2a6184cf11fc24d9c2e1f"}
Sep 16 18:59:17.069187 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:17.069150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" event={"ID":"b1ca5cbf-7961-4ba3-b39e-42e571bb266c","Type":"ContainerStarted","Data":"383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650"}
Sep 16 18:59:17.069609 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:17.069254    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 18:59:17.111682 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:17.111628    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" podStartSLOduration=1.724327641 podStartE2EDuration="4.111612435s" totalImagesPullingTime="2.387284794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:17.110311656 +0000 UTC m=+1198.898640066" watchObservedRunningTime="2026-09-16 18:59:17.111612435 +0000 UTC m=+1198.899940801"
Sep 16 18:59:18.726001 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:18.725971    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:59:18.726923 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:18.726902    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 18:59:34.440287 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.440253    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"]
Sep 16 18:59:34.444866 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.444838    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 18:59:34.453384 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.453358    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 18:59:34.458340 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.458319    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-fv78j\""
Sep 16 18:59:34.458469 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.458441    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 18:59:34.459034 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.459015    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-dktdn\""
Sep 16 18:59:34.459129 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.459094    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 18:59:34.459813 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.459790    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"]
Sep 16 18:59:34.575380 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.575336    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56\" (UID: \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 18:59:34.575380 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.575379    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2gzl\" (UniqueName: \"kubernetes.io/projected/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-kube-api-access-s2gzl\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56\" (UID: \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 18:59:34.676812 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.676770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56\" (UID: \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 18:59:34.676812 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.676812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2gzl\" (UniqueName: \"kubernetes.io/projected/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-kube-api-access-s2gzl\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56\" (UID: \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 18:59:34.679267 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.679237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56\" (UID: \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 18:59:34.688322 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.688294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2gzl\" (UniqueName: \"kubernetes.io/projected/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-kube-api-access-s2gzl\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56\" (UID: \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 18:59:34.754973 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:34.754944    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 18:59:35.040042 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:35.040017    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"]
Sep 16 18:59:35.041829 ip-10-0-142-197 kubenswrapper[2561]: W0916 18:59:35.041797    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61b0119_0caf_4f68_b550_fcc5b07d1ee5.slice/crio-2ec6699721c0e570681014bb8ecddd32d52c5767764daa1b38c5ff0a3861e7a2 WatchSource:0}: Error finding container 2ec6699721c0e570681014bb8ecddd32d52c5767764daa1b38c5ff0a3861e7a2: Status 404 returned error can't find the container with id 2ec6699721c0e570681014bb8ecddd32d52c5767764daa1b38c5ff0a3861e7a2
Sep 16 18:59:35.151507 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:35.151466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" event={"ID":"b61b0119-0caf-4f68-b550-fcc5b07d1ee5","Type":"ContainerStarted","Data":"00682e5a19bff4ce1b36bcf149e4c82d97650ae4da704edf0ccae689d9cdf08d"}
Sep 16 18:59:35.151705 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:35.151524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" event={"ID":"b61b0119-0caf-4f68-b550-fcc5b07d1ee5","Type":"ContainerStarted","Data":"2ec6699721c0e570681014bb8ecddd32d52c5767764daa1b38c5ff0a3861e7a2"}
Sep 16 18:59:35.151705 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:35.151597    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 18:59:35.226561 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:35.226503    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" podStartSLOduration=1.226481956 podStartE2EDuration="1.226481956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:35.225159531 +0000 UTC m=+1217.013487898" watchObservedRunningTime="2026-09-16 18:59:35.226481956 +0000 UTC m=+1217.014810370"
Sep 16 18:59:48.076322 ip-10-0-142-197 kubenswrapper[2561]: I0916 18:59:48.076287    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 19:00:06.158595 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:06.158543    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 19:00:08.325363 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:08.325328    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"]
Sep 16 19:00:16.158542 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:16.158456    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" podUID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:23.344623 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:23.344557    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" podUID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" containerName="main" containerID="cri-o://00682e5a19bff4ce1b36bcf149e4c82d97650ae4da704edf0ccae689d9cdf08d" gracePeriod=45
Sep 16 19:00:26.157988 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:26.157929    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" podUID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:36.158518 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:36.158464    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" podUID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:36.159245 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:36.158621    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 19:00:46.158174 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:46.158127    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" podUID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:53.524060 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:53.524022    2561 generic.go:356] "Generic (PLEG): container finished" podID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" containerID="00682e5a19bff4ce1b36bcf149e4c82d97650ae4da704edf0ccae689d9cdf08d" exitCode=0
Sep 16 19:00:53.524408 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:53.524098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" event={"ID":"b61b0119-0caf-4f68-b550-fcc5b07d1ee5","Type":"ContainerDied","Data":"00682e5a19bff4ce1b36bcf149e4c82d97650ae4da704edf0ccae689d9cdf08d"}
Sep 16 19:00:53.588750 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:53.588728    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 19:00:53.651953 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:53.651864    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-tls-certs\") pod \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\" (UID: \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\") "
Sep 16 19:00:53.652124 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:53.651961    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2gzl\" (UniqueName: \"kubernetes.io/projected/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-kube-api-access-s2gzl\") pod \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\" (UID: \"b61b0119-0caf-4f68-b550-fcc5b07d1ee5\") "
Sep 16 19:00:53.654050 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:53.654023    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b61b0119-0caf-4f68-b550-fcc5b07d1ee5" (UID: "b61b0119-0caf-4f68-b550-fcc5b07d1ee5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:53.654159 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:53.654093    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-kube-api-access-s2gzl" (OuterVolumeSpecName: "kube-api-access-s2gzl") pod "b61b0119-0caf-4f68-b550-fcc5b07d1ee5" (UID: "b61b0119-0caf-4f68-b550-fcc5b07d1ee5"). InnerVolumeSpecName "kube-api-access-s2gzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:53.752904 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:53.752865    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:00:53.752904 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:53.752895    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2gzl\" (UniqueName: \"kubernetes.io/projected/b61b0119-0caf-4f68-b550-fcc5b07d1ee5-kube-api-access-s2gzl\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:00:54.529254 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:54.529219    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"
Sep 16 19:00:54.529254 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:54.529248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56" event={"ID":"b61b0119-0caf-4f68-b550-fcc5b07d1ee5","Type":"ContainerDied","Data":"2ec6699721c0e570681014bb8ecddd32d52c5767764daa1b38c5ff0a3861e7a2"}
Sep 16 19:00:54.529903 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:54.529304    2561 scope.go:117] "RemoveContainer" containerID="00682e5a19bff4ce1b36bcf149e4c82d97650ae4da704edf0ccae689d9cdf08d"
Sep 16 19:00:54.557194 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:54.557162    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"]
Sep 16 19:00:54.563713 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:54.563688    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheq2q56"]
Sep 16 19:00:54.750060 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:00:54.750029    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" path="/var/lib/kubelet/pods/b61b0119-0caf-4f68-b550-fcc5b07d1ee5/volumes"
Sep 16 19:01:01.691149 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.691113    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"]
Sep 16 19:01:01.691534 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.691464    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" containerName="main"
Sep 16 19:01:01.691534 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.691477    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" containerName="main"
Sep 16 19:01:01.691670 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.691536    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61b0119-0caf-4f68-b550-fcc5b07d1ee5" containerName="main"
Sep 16 19:01:01.695908 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.695888    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:01.703072 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.703038    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-9ttnw\""
Sep 16 19:01:01.703191 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.703089    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 19:01:01.703191 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.703159    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 19:01:01.703400 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.703384    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-6l6g9\""
Sep 16 19:01:01.703505 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.703419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 19:01:01.720065 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.720036    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"]
Sep 16 19:01:01.826526 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.826483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0ee487-1215-4ac6-90b0-8af58a7a681c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs\" (UID: \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:01.826526 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.826527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n5sw\" (UniqueName: \"kubernetes.io/projected/3d0ee487-1215-4ac6-90b0-8af58a7a681c-kube-api-access-7n5sw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs\" (UID: \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:01.927229 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.927190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0ee487-1215-4ac6-90b0-8af58a7a681c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs\" (UID: \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:01.927450 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.927243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n5sw\" (UniqueName: \"kubernetes.io/projected/3d0ee487-1215-4ac6-90b0-8af58a7a681c-kube-api-access-7n5sw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs\" (UID: \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:01.929926 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.929898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0ee487-1215-4ac6-90b0-8af58a7a681c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs\" (UID: \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:01.937033 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:01.937007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n5sw\" (UniqueName: \"kubernetes.io/projected/3d0ee487-1215-4ac6-90b0-8af58a7a681c-kube-api-access-7n5sw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs\" (UID: \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:02.006282 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:02.006239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:02.284868 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:02.284781    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"]
Sep 16 19:01:02.287037 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:01:02.287011    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0ee487_1215_4ac6_90b0_8af58a7a681c.slice/crio-d640f498a1d200ea4fef509ad961c94839a20d24dbf26dd2a96bb8e87dbb0d01 WatchSource:0}: Error finding container d640f498a1d200ea4fef509ad961c94839a20d24dbf26dd2a96bb8e87dbb0d01: Status 404 returned error can't find the container with id d640f498a1d200ea4fef509ad961c94839a20d24dbf26dd2a96bb8e87dbb0d01
Sep 16 19:01:02.568176 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:02.568068    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" event={"ID":"3d0ee487-1215-4ac6-90b0-8af58a7a681c","Type":"ContainerStarted","Data":"f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e"}
Sep 16 19:01:02.568176 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:02.568123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" event={"ID":"3d0ee487-1215-4ac6-90b0-8af58a7a681c","Type":"ContainerStarted","Data":"d640f498a1d200ea4fef509ad961c94839a20d24dbf26dd2a96bb8e87dbb0d01"}
Sep 16 19:01:02.568176 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:02.568145    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:02.597084 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:02.597028    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" podStartSLOduration=1.5970080260000001 podStartE2EDuration="1.597008026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:02.595333787 +0000 UTC m=+1304.383662155" watchObservedRunningTime="2026-09-16 19:01:02.597008026 +0000 UTC m=+1304.385336439"
Sep 16 19:01:22.271728 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:22.271689    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"]
Sep 16 19:01:28.076137 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:28.076092    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" podUID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:33.574994 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:33.574965    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:01:35.646922 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:35.646892    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"]
Sep 16 19:01:37.291484 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:37.291444    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" podUID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" containerName="main" containerID="cri-o://383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650" gracePeriod=45
Sep 16 19:01:38.076102 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:38.076058    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" podUID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:43.575684 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:43.575594    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" podUID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:48.076099 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:48.076059    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" podUID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:48.076484 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:48.076168    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 19:01:50.666332 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:50.666281    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" podUID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" containerName="main" containerID="cri-o://f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e" gracePeriod=45
Sep 16 19:01:53.575354 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:53.575312    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" podUID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:58.076230 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:01:58.076192    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" podUID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:03.575342 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:03.575305    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" podUID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:03.575810 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:03.575424    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:02:07.546843 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.546811    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 19:02:07.708415 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.708316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdmfn\" (UniqueName: \"kubernetes.io/projected/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-kube-api-access-qdmfn\") pod \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\" (UID: \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\") "
Sep 16 19:02:07.708415 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.708374    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-tls-certs\") pod \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\" (UID: \"b1ca5cbf-7961-4ba3-b39e-42e571bb266c\") "
Sep 16 19:02:07.710637 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.710604    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1ca5cbf-7961-4ba3-b39e-42e571bb266c" (UID: "b1ca5cbf-7961-4ba3-b39e-42e571bb266c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:07.710637 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.710618    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-kube-api-access-qdmfn" (OuterVolumeSpecName: "kube-api-access-qdmfn") pod "b1ca5cbf-7961-4ba3-b39e-42e571bb266c" (UID: "b1ca5cbf-7961-4ba3-b39e-42e571bb266c"). InnerVolumeSpecName "kube-api-access-qdmfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:07.809076 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.809041    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdmfn\" (UniqueName: \"kubernetes.io/projected/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-kube-api-access-qdmfn\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:02:07.809076 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.809071    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ca5cbf-7961-4ba3-b39e-42e571bb266c-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:02:07.874652 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.874615    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" containerID="383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650" exitCode=0
Sep 16 19:02:07.874843 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.874660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" event={"ID":"b1ca5cbf-7961-4ba3-b39e-42e571bb266c","Type":"ContainerDied","Data":"383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650"}
Sep 16 19:02:07.874843 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.874674    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"
Sep 16 19:02:07.874843 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.874703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n" event={"ID":"b1ca5cbf-7961-4ba3-b39e-42e571bb266c","Type":"ContainerDied","Data":"90711457780afa47cde18c326ca548c61642c1018dc2a6184cf11fc24d9c2e1f"}
Sep 16 19:02:07.874843 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.874729    2561 scope.go:117] "RemoveContainer" containerID="383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650"
Sep 16 19:02:07.883669 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.883647    2561 scope.go:117] "RemoveContainer" containerID="383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650"
Sep 16 19:02:07.883957 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:02:07.883939    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650\": container with ID starting with 383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650 not found: ID does not exist" containerID="383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650"
Sep 16 19:02:07.884001 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.883966    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650"} err="failed to get container status \"383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650\": rpc error: code = NotFound desc = could not find container \"383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650\": container with ID starting with 383e89edd11e1cbe99d56f468b42bf8eeee3e3cb68bebd5faa4c025b322a9650 not found: ID does not exist"
Sep 16 19:02:07.897980 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.897951    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"]
Sep 16 19:02:07.902310 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:07.902284    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c8z97n"]
Sep 16 19:02:08.749087 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:08.749053    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" path="/var/lib/kubelet/pods/b1ca5cbf-7961-4ba3-b39e-42e571bb266c/volumes"
Sep 16 19:02:13.575139 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:13.575101    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" podUID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:20.914810 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:20.914784    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:02:20.941426 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:20.941338    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" containerID="f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e" exitCode=0
Sep 16 19:02:20.941426 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:20.941381    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"
Sep 16 19:02:20.941643 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:20.941416    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" event={"ID":"3d0ee487-1215-4ac6-90b0-8af58a7a681c","Type":"ContainerDied","Data":"f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e"}
Sep 16 19:02:20.941643 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:20.941474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs" event={"ID":"3d0ee487-1215-4ac6-90b0-8af58a7a681c","Type":"ContainerDied","Data":"d640f498a1d200ea4fef509ad961c94839a20d24dbf26dd2a96bb8e87dbb0d01"}
Sep 16 19:02:20.941643 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:20.941502    2561 scope.go:117] "RemoveContainer" containerID="f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e"
Sep 16 19:02:20.949804 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:20.949782    2561 scope.go:117] "RemoveContainer" containerID="f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e"
Sep 16 19:02:20.950108 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:02:20.950090    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e\": container with ID starting with f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e not found: ID does not exist" containerID="f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e"
Sep 16 19:02:20.950164 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:20.950116    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e"} err="failed to get container status \"f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e\": rpc error: code = NotFound desc = could not find container \"f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e\": container with ID starting with f4300a68065d062ac85aaa77677ca3d39f764ef59a2e2dc91a3f8cfa95aafd8e not found: ID does not exist"
Sep 16 19:02:21.011538 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:21.011497    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0ee487-1215-4ac6-90b0-8af58a7a681c-tls-certs\") pod \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\" (UID: \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\") "
Sep 16 19:02:21.011755 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:21.011563    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n5sw\" (UniqueName: \"kubernetes.io/projected/3d0ee487-1215-4ac6-90b0-8af58a7a681c-kube-api-access-7n5sw\") pod \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\" (UID: \"3d0ee487-1215-4ac6-90b0-8af58a7a681c\") "
Sep 16 19:02:21.013827 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:21.013801    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0ee487-1215-4ac6-90b0-8af58a7a681c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d0ee487-1215-4ac6-90b0-8af58a7a681c" (UID: "3d0ee487-1215-4ac6-90b0-8af58a7a681c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:21.013827 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:21.013829    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0ee487-1215-4ac6-90b0-8af58a7a681c-kube-api-access-7n5sw" (OuterVolumeSpecName: "kube-api-access-7n5sw") pod "3d0ee487-1215-4ac6-90b0-8af58a7a681c" (UID: "3d0ee487-1215-4ac6-90b0-8af58a7a681c"). InnerVolumeSpecName "kube-api-access-7n5sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:21.112908 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:21.112873    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0ee487-1215-4ac6-90b0-8af58a7a681c-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:02:21.112908 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:21.112903    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n5sw\" (UniqueName: \"kubernetes.io/projected/3d0ee487-1215-4ac6-90b0-8af58a7a681c-kube-api-access-7n5sw\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:02:21.264093 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:21.264053    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"]
Sep 16 19:02:21.267357 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:21.267330    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c96hxfs"]
Sep 16 19:02:22.749725 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:22.749687    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" path="/var/lib/kubelet/pods/3d0ee487-1215-4ac6-90b0-8af58a7a681c/volumes"
Sep 16 19:02:44.577672 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.577634    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"]
Sep 16 19:02:44.578251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.577955    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" containerName="main"
Sep 16 19:02:44.578251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.577966    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" containerName="main"
Sep 16 19:02:44.578251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.577983    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" containerName="main"
Sep 16 19:02:44.578251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.577989    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" containerName="main"
Sep 16 19:02:44.578251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.578054    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d0ee487-1215-4ac6-90b0-8af58a7a681c" containerName="main"
Sep 16 19:02:44.578251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.578061    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ca5cbf-7961-4ba3-b39e-42e571bb266c" containerName="main"
Sep 16 19:02:44.582866 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.582846    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.586212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.586187    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-c2qsv\""
Sep 16 19:02:44.586335 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.586187    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:02:44.586335 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.586233    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:02:44.586335 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.586306    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:02:44.593473 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.593449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"]
Sep 16 19:02:44.730165 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.730126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.730165 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.730170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.730388 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.730223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.730388 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.730252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.730388 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.730293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ngwz\" (UniqueName: \"kubernetes.io/projected/30796f42-4ed9-4b87-ab71-9951cd9c874a-kube-api-access-9ngwz\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.730388 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.730340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30796f42-4ed9-4b87-ab71-9951cd9c874a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.814386 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.814355    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"]
Sep 16 19:02:44.817715 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.817698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:02:44.820846 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.820824    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-2dkzp\""
Sep 16 19:02:44.829898 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.829841    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"]
Sep 16 19:02:44.831097 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.831073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.831205 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.831116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.831205 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.831151    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.831205 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.831176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.831389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.831225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ngwz\" (UniqueName: \"kubernetes.io/projected/30796f42-4ed9-4b87-ab71-9951cd9c874a-kube-api-access-9ngwz\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.831389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.831325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30796f42-4ed9-4b87-ab71-9951cd9c874a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.831543 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.831520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.831780 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.831756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.831862 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.831803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.833515 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.833486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.833824 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.833807    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30796f42-4ed9-4b87-ab71-9951cd9c874a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.840810 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.840791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ngwz\" (UniqueName: \"kubernetes.io/projected/30796f42-4ed9-4b87-ab71-9951cd9c874a-kube-api-access-9ngwz\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.893939 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.893892    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:44.932604 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.932547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqkr\" (UniqueName: \"kubernetes.io/projected/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-kube-api-access-7wqkr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff\" (UID: \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:02:44.932885 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:44.932863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff\" (UID: \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:02:45.034405 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:45.034353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqkr\" (UniqueName: \"kubernetes.io/projected/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-kube-api-access-7wqkr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff\" (UID: \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:02:45.034405 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:45.034411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff\" (UID: \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:02:45.036987 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:45.036964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff\" (UID: \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:02:45.043718 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:45.043693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqkr\" (UniqueName: \"kubernetes.io/projected/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-kube-api-access-7wqkr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff\" (UID: \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:02:45.127871 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:45.127779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:02:45.171641 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:45.171606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"]
Sep 16 19:02:45.175006 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:02:45.174980    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30796f42_4ed9_4b87_ab71_9951cd9c874a.slice/crio-3d4f6f142f8032f3bb4ec39eb04f286ec1e9910b40d32b17a5824be987f01e1f WatchSource:0}: Error finding container 3d4f6f142f8032f3bb4ec39eb04f286ec1e9910b40d32b17a5824be987f01e1f: Status 404 returned error can't find the container with id 3d4f6f142f8032f3bb4ec39eb04f286ec1e9910b40d32b17a5824be987f01e1f
Sep 16 19:02:45.403051 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:45.402964    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"]
Sep 16 19:02:45.406275 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:02:45.406247    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21ef5b6_5b59_46ec_9c4f_b9143c1deb37.slice/crio-ad74c643f499e209e51ff8001792e1ae5521592179a213ca37f26ad17057177c WatchSource:0}: Error finding container ad74c643f499e209e51ff8001792e1ae5521592179a213ca37f26ad17057177c: Status 404 returned error can't find the container with id ad74c643f499e209e51ff8001792e1ae5521592179a213ca37f26ad17057177c
Sep 16 19:02:46.057958 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:46.057918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff" event={"ID":"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37","Type":"ContainerStarted","Data":"45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b"}
Sep 16 19:02:46.058395 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:46.057980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff" event={"ID":"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37","Type":"ContainerStarted","Data":"ad74c643f499e209e51ff8001792e1ae5521592179a213ca37f26ad17057177c"}
Sep 16 19:02:46.058395 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:46.058009    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:02:46.059677 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:46.059651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx" event={"ID":"30796f42-4ed9-4b87-ab71-9951cd9c874a","Type":"ContainerStarted","Data":"10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2"}
Sep 16 19:02:46.059805 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:46.059690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx" event={"ID":"30796f42-4ed9-4b87-ab71-9951cd9c874a","Type":"ContainerStarted","Data":"3d4f6f142f8032f3bb4ec39eb04f286ec1e9910b40d32b17a5824be987f01e1f"}
Sep 16 19:02:46.078148 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:46.078096    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff" podStartSLOduration=2.078077334 podStartE2EDuration="2.078077334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:46.075887894 +0000 UTC m=+1407.864216261" watchObservedRunningTime="2026-09-16 19:02:46.078077334 +0000 UTC m=+1407.866405705"
Sep 16 19:02:46.095782 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:46.095729    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx" podStartSLOduration=2.095708421 podStartE2EDuration="2.095708421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:46.094894922 +0000 UTC m=+1407.883223289" watchObservedRunningTime="2026-09-16 19:02:46.095708421 +0000 UTC m=+1407.884036832"
Sep 16 19:02:46.894311 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:46.894280    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:54.894476 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:54.894431    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:54.907263 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:54.907235    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:02:55.118240 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:02:55.118200    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:03:17.066829 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:17.066746    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:03:18.654237 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:18.654197    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"]
Sep 16 19:03:18.664855 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:18.664822    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"]
Sep 16 19:03:18.673532 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:03:18.673490    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:03:18.673532 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:18.673532 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:18.673532 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2" cmd=["/bin/sleep","15"]
Sep 16 19:03:18.673780 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:18.673547    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx" message="" err=<
Sep 16 19:03:18.673780 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:18.673780 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:18.673780 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 19:03:18.673780 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:03:18.673559    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:03:18.673780 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:18.673780 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:18.673780 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx" podUID="30796f42-4ed9-4b87-ab71-9951cd9c874a" containerName="main" containerID="cri-o://10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2"
Sep 16 19:03:18.673780 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:18.673616    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx" podUID="30796f42-4ed9-4b87-ab71-9951cd9c874a" containerName="main" containerID="cri-o://10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2" gracePeriod=60
Sep 16 19:03:19.015540 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.015516    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:03:19.130407 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130355    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-model-cache\") pod \"30796f42-4ed9-4b87-ab71-9951cd9c874a\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") "
Sep 16 19:03:19.130653 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130424    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-dshm\") pod \"30796f42-4ed9-4b87-ab71-9951cd9c874a\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") "
Sep 16 19:03:19.130653 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130448    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-home\") pod \"30796f42-4ed9-4b87-ab71-9951cd9c874a\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") "
Sep 16 19:03:19.130653 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130511    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ngwz\" (UniqueName: \"kubernetes.io/projected/30796f42-4ed9-4b87-ab71-9951cd9c874a-kube-api-access-9ngwz\") pod \"30796f42-4ed9-4b87-ab71-9951cd9c874a\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") "
Sep 16 19:03:19.130653 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130563    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30796f42-4ed9-4b87-ab71-9951cd9c874a-tls-certs\") pod \"30796f42-4ed9-4b87-ab71-9951cd9c874a\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") "
Sep 16 19:03:19.130653 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130625    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-tmp-dir\") pod \"30796f42-4ed9-4b87-ab71-9951cd9c874a\" (UID: \"30796f42-4ed9-4b87-ab71-9951cd9c874a\") "
Sep 16 19:03:19.130933 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130728    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-model-cache" (OuterVolumeSpecName: "model-cache") pod "30796f42-4ed9-4b87-ab71-9951cd9c874a" (UID: "30796f42-4ed9-4b87-ab71-9951cd9c874a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:19.130933 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-home" (OuterVolumeSpecName: "home") pod "30796f42-4ed9-4b87-ab71-9951cd9c874a" (UID: "30796f42-4ed9-4b87-ab71-9951cd9c874a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:19.131026 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130935    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:03:19.131026 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130936    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30796f42-4ed9-4b87-ab71-9951cd9c874a" (UID: "30796f42-4ed9-4b87-ab71-9951cd9c874a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:19.131026 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.130958    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:03:19.132866 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.132840    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-dshm" (OuterVolumeSpecName: "dshm") pod "30796f42-4ed9-4b87-ab71-9951cd9c874a" (UID: "30796f42-4ed9-4b87-ab71-9951cd9c874a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:19.132993 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.132960    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30796f42-4ed9-4b87-ab71-9951cd9c874a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30796f42-4ed9-4b87-ab71-9951cd9c874a" (UID: "30796f42-4ed9-4b87-ab71-9951cd9c874a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:19.132993 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.132977    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30796f42-4ed9-4b87-ab71-9951cd9c874a-kube-api-access-9ngwz" (OuterVolumeSpecName: "kube-api-access-9ngwz") pod "30796f42-4ed9-4b87-ab71-9951cd9c874a" (UID: "30796f42-4ed9-4b87-ab71-9951cd9c874a"). InnerVolumeSpecName "kube-api-access-9ngwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:19.214803 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.214717    2561 generic.go:356] "Generic (PLEG): container finished" podID="30796f42-4ed9-4b87-ab71-9951cd9c874a" containerID="10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2" exitCode=0
Sep 16 19:03:19.214803 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.214765    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"
Sep 16 19:03:19.215039 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.214801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx" event={"ID":"30796f42-4ed9-4b87-ab71-9951cd9c874a","Type":"ContainerDied","Data":"10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2"}
Sep 16 19:03:19.215039 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.214868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx" event={"ID":"30796f42-4ed9-4b87-ab71-9951cd9c874a","Type":"ContainerDied","Data":"3d4f6f142f8032f3bb4ec39eb04f286ec1e9910b40d32b17a5824be987f01e1f"}
Sep 16 19:03:19.215039 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.214898    2561 scope.go:117] "RemoveContainer" containerID="10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2"
Sep 16 19:03:19.225863 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.225845    2561 scope.go:117] "RemoveContainer" containerID="10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2"
Sep 16 19:03:19.226187 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:03:19.226168    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2\": container with ID starting with 10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2 not found: ID does not exist" containerID="10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2"
Sep 16 19:03:19.226252 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.226196    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2"} err="failed to get container status \"10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2\": rpc error: code = NotFound desc = could not find container \"10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2\": container with ID starting with 10063d6fc1e9890dc46d86868f0273bb9dfb599f594ab15187332a69c7839be2 not found: ID does not exist"
Sep 16 19:03:19.232144 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.232123    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ngwz\" (UniqueName: \"kubernetes.io/projected/30796f42-4ed9-4b87-ab71-9951cd9c874a-kube-api-access-9ngwz\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:03:19.232144 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.232144    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30796f42-4ed9-4b87-ab71-9951cd9c874a-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:03:19.232264 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.232174    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:03:19.232264 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.232183    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30796f42-4ed9-4b87-ab71-9951cd9c874a-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:03:19.237335 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.237314    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"]
Sep 16 19:03:19.240994 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:19.240975    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zrlsx"]
Sep 16 19:03:20.749167 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:20.749129    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30796f42-4ed9-4b87-ab71-9951cd9c874a" path="/var/lib/kubelet/pods/30796f42-4ed9-4b87-ab71-9951cd9c874a/volumes"
Sep 16 19:03:27.066589 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:27.066531    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff" podUID="a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:30.498142 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.498107    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 19:03:30.498403 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:03:30.498145    2561 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff" podUID="a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" containerName="main" containerID="cri-o://45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b"
Sep 16 19:03:30.498403 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.498200    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff" podUID="a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" containerName="main" containerID="cri-o://45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b" gracePeriod=49
Sep 16 19:03:30.634477 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.634451    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff_a21ef5b6-5b59-46ec-9c4f-b9143c1deb37/main/0.log"
Sep 16 19:03:30.636829 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.636813    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:03:30.736636 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.736603    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wqkr\" (UniqueName: \"kubernetes.io/projected/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-kube-api-access-7wqkr\") pod \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\" (UID: \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\") "
Sep 16 19:03:30.736809 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.736658    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-tls-certs\") pod \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\" (UID: \"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37\") "
Sep 16 19:03:30.738724 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.738699    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" (UID: "a21ef5b6-5b59-46ec-9c4f-b9143c1deb37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:30.738724 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.738701    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-kube-api-access-7wqkr" (OuterVolumeSpecName: "kube-api-access-7wqkr") pod "a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" (UID: "a21ef5b6-5b59-46ec-9c4f-b9143c1deb37"). InnerVolumeSpecName "kube-api-access-7wqkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:30.837759 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.837722    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wqkr\" (UniqueName: \"kubernetes.io/projected/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-kube-api-access-7wqkr\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:03:30.837759 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:30.837750    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:03:31.268578 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.268548    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff_a21ef5b6-5b59-46ec-9c4f-b9143c1deb37/main/0.log"
Sep 16 19:03:31.268784 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.268604    2561 generic.go:356] "Generic (PLEG): container finished" podID="a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" containerID="45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b" exitCode=1
Sep 16 19:03:31.268784 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.268637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff" event={"ID":"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37","Type":"ContainerDied","Data":"45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b"}
Sep 16 19:03:31.268784 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.268704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff" event={"ID":"a21ef5b6-5b59-46ec-9c4f-b9143c1deb37","Type":"ContainerDied","Data":"ad74c643f499e209e51ff8001792e1ae5521592179a213ca37f26ad17057177c"}
Sep 16 19:03:31.268784 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.268652    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"
Sep 16 19:03:31.268784 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.268734    2561 scope.go:117] "RemoveContainer" containerID="45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b"
Sep 16 19:03:31.277521 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.277503    2561 scope.go:117] "RemoveContainer" containerID="45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b"
Sep 16 19:03:31.277829 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:03:31.277805    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b\": container with ID starting with 45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b not found: ID does not exist" containerID="45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b"
Sep 16 19:03:31.277897 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.277832    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b"} err="failed to get container status \"45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b\": rpc error: code = NotFound desc = could not find container \"45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b\": container with ID starting with 45a2f80e99859e6fc3af74cbbabad60e33c75486fbf23d0978fdf8ff1d71819b not found: ID does not exist"
Sep 16 19:03:31.286946 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.286925    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"]
Sep 16 19:03:31.291591 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:31.291556    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5pskff"]
Sep 16 19:03:32.749147 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:03:32.749102    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" path="/var/lib/kubelet/pods/a21ef5b6-5b59-46ec-9c4f-b9143c1deb37/volumes"
Sep 16 19:04:18.759495 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:18.759467    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:04:18.759983 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:18.759711    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:04:20.613139 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.613105    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"]
Sep 16 19:04:20.613503 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.613438    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" containerName="main"
Sep 16 19:04:20.613503 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.613451    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" containerName="main"
Sep 16 19:04:20.613503 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.613468    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30796f42-4ed9-4b87-ab71-9951cd9c874a" containerName="main"
Sep 16 19:04:20.613503 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.613474    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="30796f42-4ed9-4b87-ab71-9951cd9c874a" containerName="main"
Sep 16 19:04:20.613671 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.613529    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a21ef5b6-5b59-46ec-9c4f-b9143c1deb37" containerName="main"
Sep 16 19:04:20.613671 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.613537    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="30796f42-4ed9-4b87-ab71-9951cd9c874a" containerName="main"
Sep 16 19:04:20.616335 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.616318    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.618719 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.618690    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 19:04:20.619623 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.619603    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 19:04:20.619736 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.619610    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-55n52\""
Sep 16 19:04:20.619736 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.619673    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 19:04:20.628590 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.628550    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"]
Sep 16 19:04:20.769013 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.768976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.769013 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.769015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.769248 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.769053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwqb\" (UniqueName: \"kubernetes.io/projected/bc7d7439-64d1-4cd4-b8ab-885109c32f27-kube-api-access-frwqb\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.769248 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.769126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.769248 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.769162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.769248 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.769187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.870249 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.870156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwqb\" (UniqueName: \"kubernetes.io/projected/bc7d7439-64d1-4cd4-b8ab-885109c32f27-kube-api-access-frwqb\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.870249 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.870208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.870249 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.870230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.870249 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.870252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.870621 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.870287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.870621 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.870302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.870777 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.870742    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.870777 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.870763    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.870930 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.870790    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.872559 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.872527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.872840 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.872824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.878415 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.878391    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwqb\" (UniqueName: \"kubernetes.io/projected/bc7d7439-64d1-4cd4-b8ab-885109c32f27-kube-api-access-frwqb\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nzrm6\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:20.925817 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:20.925780    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:21.207985 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:21.207956    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"]
Sep 16 19:04:21.209807 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:04:21.209779    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7d7439_64d1_4cd4_b8ab_885109c32f27.slice/crio-e9530b0d8f143c8061d2612da5da420cebc97dd5d30abff15a6dfc54df4539cb WatchSource:0}: Error finding container e9530b0d8f143c8061d2612da5da420cebc97dd5d30abff15a6dfc54df4539cb: Status 404 returned error can't find the container with id e9530b0d8f143c8061d2612da5da420cebc97dd5d30abff15a6dfc54df4539cb
Sep 16 19:04:21.502421 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:21.502386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6" event={"ID":"bc7d7439-64d1-4cd4-b8ab-885109c32f27","Type":"ContainerStarted","Data":"a34d623ab09e3d48b8a2f3f73e323c446595a5f61555c98b9b513c16bd83da4c"}
Sep 16 19:04:21.502624 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:21.502467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6" event={"ID":"bc7d7439-64d1-4cd4-b8ab-885109c32f27","Type":"ContainerStarted","Data":"e9530b0d8f143c8061d2612da5da420cebc97dd5d30abff15a6dfc54df4539cb"}
Sep 16 19:04:21.527738 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:21.527684    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6" podStartSLOduration=1.527667465 podStartE2EDuration="1.527667465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:21.526325516 +0000 UTC m=+1503.314653881" watchObservedRunningTime="2026-09-16 19:04:21.527667465 +0000 UTC m=+1503.315995830"
Sep 16 19:04:21.926003 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:21.925910    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:30.926759 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:30.926713    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:30.939295 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:30.939267    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:31.556830 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:31.556795    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:54.349108 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.349022    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"]
Sep 16 19:04:54.359285 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:04:54.359248    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:04:54.359285 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:54.359285 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:54.359285 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="a34d623ab09e3d48b8a2f3f73e323c446595a5f61555c98b9b513c16bd83da4c" cmd=["/bin/sleep","15"]
Sep 16 19:04:54.359650 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.359303    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6" message="" err=<
Sep 16 19:04:54.359650 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:54.359650 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:54.359650 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 19:04:54.359650 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:04:54.359316    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:04:54.359650 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:54.359650 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:54.359650 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6" podUID="bc7d7439-64d1-4cd4-b8ab-885109c32f27" containerName="main" containerID="cri-o://a34d623ab09e3d48b8a2f3f73e323c446595a5f61555c98b9b513c16bd83da4c"
Sep 16 19:04:54.359650 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.359356    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6" podUID="bc7d7439-64d1-4cd4-b8ab-885109c32f27" containerName="main" containerID="cri-o://a34d623ab09e3d48b8a2f3f73e323c446595a5f61555c98b9b513c16bd83da4c" gracePeriod=60
Sep 16 19:04:54.546639 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.546556    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6" podUID="bc7d7439-64d1-4cd4-b8ab-885109c32f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 16 19:04:54.654337 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.654304    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc7d7439-64d1-4cd4-b8ab-885109c32f27" containerID="a34d623ab09e3d48b8a2f3f73e323c446595a5f61555c98b9b513c16bd83da4c" exitCode=0
Sep 16 19:04:54.654490 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.654360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6" event={"ID":"bc7d7439-64d1-4cd4-b8ab-885109c32f27","Type":"ContainerDied","Data":"a34d623ab09e3d48b8a2f3f73e323c446595a5f61555c98b9b513c16bd83da4c"}
Sep 16 19:04:54.712188 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.712161    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:54.754497 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.754462    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frwqb\" (UniqueName: \"kubernetes.io/projected/bc7d7439-64d1-4cd4-b8ab-885109c32f27-kube-api-access-frwqb\") pod \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") "
Sep 16 19:04:54.754711 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.754518    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-dshm\") pod \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") "
Sep 16 19:04:54.754711 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.754534    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tmp-dir\") pod \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") "
Sep 16 19:04:54.754711 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.754555    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-model-cache\") pod \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") "
Sep 16 19:04:54.754711 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.754652    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-home\") pod \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") "
Sep 16 19:04:54.754711 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.754703    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tls-certs\") pod \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\" (UID: \"bc7d7439-64d1-4cd4-b8ab-885109c32f27\") "
Sep 16 19:04:54.755041 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.754879    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc7d7439-64d1-4cd4-b8ab-885109c32f27" (UID: "bc7d7439-64d1-4cd4-b8ab-885109c32f27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:54.755041 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.754966    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:04:54.755251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.755205    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc7d7439-64d1-4cd4-b8ab-885109c32f27" (UID: "bc7d7439-64d1-4cd4-b8ab-885109c32f27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:54.755460 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.755436    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-home" (OuterVolumeSpecName: "home") pod "bc7d7439-64d1-4cd4-b8ab-885109c32f27" (UID: "bc7d7439-64d1-4cd4-b8ab-885109c32f27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:54.757306 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.757276    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc7d7439-64d1-4cd4-b8ab-885109c32f27" (UID: "bc7d7439-64d1-4cd4-b8ab-885109c32f27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:54.757416 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.757309    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-dshm" (OuterVolumeSpecName: "dshm") pod "bc7d7439-64d1-4cd4-b8ab-885109c32f27" (UID: "bc7d7439-64d1-4cd4-b8ab-885109c32f27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:54.757416 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.757322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7d7439-64d1-4cd4-b8ab-885109c32f27-kube-api-access-frwqb" (OuterVolumeSpecName: "kube-api-access-frwqb") pod "bc7d7439-64d1-4cd4-b8ab-885109c32f27" (UID: "bc7d7439-64d1-4cd4-b8ab-885109c32f27"). InnerVolumeSpecName "kube-api-access-frwqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:54.855743 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.855705    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:04:54.855743 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.855738    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:04:54.855743 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.855749    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7d7439-64d1-4cd4-b8ab-885109c32f27-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:04:54.855743 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.855756    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7d7439-64d1-4cd4-b8ab-885109c32f27-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:04:54.856067 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:54.855765    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwqb\" (UniqueName: \"kubernetes.io/projected/bc7d7439-64d1-4cd4-b8ab-885109c32f27-kube-api-access-frwqb\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:04:55.659501 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:55.659462    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"
Sep 16 19:04:55.659501 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:55.659487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6" event={"ID":"bc7d7439-64d1-4cd4-b8ab-885109c32f27","Type":"ContainerDied","Data":"e9530b0d8f143c8061d2612da5da420cebc97dd5d30abff15a6dfc54df4539cb"}
Sep 16 19:04:55.660030 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:55.659578    2561 scope.go:117] "RemoveContainer" containerID="a34d623ab09e3d48b8a2f3f73e323c446595a5f61555c98b9b513c16bd83da4c"
Sep 16 19:04:55.681707 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:55.681675    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"]
Sep 16 19:04:55.685837 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:55.685811    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nzrm6"]
Sep 16 19:04:56.749777 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:04:56.749742    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7d7439-64d1-4cd4-b8ab-885109c32f27" path="/var/lib/kubelet/pods/bc7d7439-64d1-4cd4-b8ab-885109c32f27/volumes"
Sep 16 19:06:16.795443 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.795356    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"]
Sep 16 19:06:16.795874 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.795730    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7d7439-64d1-4cd4-b8ab-885109c32f27" containerName="main"
Sep 16 19:06:16.795874 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.795742    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7d7439-64d1-4cd4-b8ab-885109c32f27" containerName="main"
Sep 16 19:06:16.795874 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.795801    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7d7439-64d1-4cd4-b8ab-885109c32f27" containerName="main"
Sep 16 19:06:16.799223 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.799199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.802598 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.802562    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:06:16.802734 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.802599    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:06:16.802734 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.802600    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-kss72\""
Sep 16 19:06:16.802734 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.802606    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:06:16.802952 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.802938    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-dprcq\""
Sep 16 19:06:16.814542 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.814513    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"]
Sep 16 19:06:16.851046 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.851010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.851046 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.851045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40395d54-188c-421d-8d22-9d891274d3b8-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.851289 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.851118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.851289 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.851160    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.851289 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.851248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktj6\" (UniqueName: \"kubernetes.io/projected/40395d54-188c-421d-8d22-9d891274d3b8-kube-api-access-gktj6\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.851289 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.851285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.952705 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.952651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.952705 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.952722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.952969 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.952738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40395d54-188c-421d-8d22-9d891274d3b8-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.952969 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.952766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.952969 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.952802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.952969 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.952937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktj6\" (UniqueName: \"kubernetes.io/projected/40395d54-188c-421d-8d22-9d891274d3b8-kube-api-access-gktj6\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.953189 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.953093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.953189 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.953105    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.953303 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.953183    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.954964 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.954943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.955240 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.955222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40395d54-188c-421d-8d22-9d891274d3b8-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:16.969770 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:16.969738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktj6\" (UniqueName: \"kubernetes.io/projected/40395d54-188c-421d-8d22-9d891274d3b8-kube-api-access-gktj6\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:17.110187 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:17.110099    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:17.395095 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:17.395064    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"]
Sep 16 19:06:17.397209 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:06:17.397183    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40395d54_188c_421d_8d22_9d891274d3b8.slice/crio-59d5e5c0e0f42fdaa2a34e61df5fd83684c5624aeb5f907f343cccd2e0ffdd3e WatchSource:0}: Error finding container 59d5e5c0e0f42fdaa2a34e61df5fd83684c5624aeb5f907f343cccd2e0ffdd3e: Status 404 returned error can't find the container with id 59d5e5c0e0f42fdaa2a34e61df5fd83684c5624aeb5f907f343cccd2e0ffdd3e
Sep 16 19:06:17.399214 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:17.399197    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:06:18.021788 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:18.021747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" event={"ID":"40395d54-188c-421d-8d22-9d891274d3b8","Type":"ContainerStarted","Data":"59d5e5c0e0f42fdaa2a34e61df5fd83684c5624aeb5f907f343cccd2e0ffdd3e"}
Sep 16 19:06:20.034704 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:20.034665    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" event={"ID":"40395d54-188c-421d-8d22-9d891274d3b8","Type":"ContainerStarted","Data":"f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1"}
Sep 16 19:06:20.035108 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:20.034743    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:21.041267 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:21.041228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" event={"ID":"40395d54-188c-421d-8d22-9d891274d3b8","Type":"ContainerStarted","Data":"7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208"}
Sep 16 19:06:21.075912 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:21.075853    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" podStartSLOduration=3.517414467 podStartE2EDuration="5.075834497s" totalImagesPullingTime="1.55842003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:21.07446019 +0000 UTC m=+1622.862788568" watchObservedRunningTime="2026-09-16 19:06:21.075834497 +0000 UTC m=+1622.864162862"
Sep 16 19:06:21.110862 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:21.110833    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:27.110300 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:27.110255    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:27.122527 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:27.122496    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:28.085095 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:28.085062    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:06:29.090981 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:06:29.090952    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:09:03.682121 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:03.682085    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"]
Sep 16 19:09:03.691311 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:09:03.691270    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:09:03.691311 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:03.691311 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:03.691311 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208" cmd=["/bin/sleep","15"]
Sep 16 19:09:03.691598 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:03.691333    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" message="" err=<
Sep 16 19:09:03.691598 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:03.691598 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:03.691598 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 19:09:03.691598 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:09:03.691347    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:09:03.691598 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:03.691598 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:03.691598 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="main" containerID="cri-o://7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208"
Sep 16 19:09:03.691598 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:03.691391    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="main" containerID="cri-o://7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208" gracePeriod=60
Sep 16 19:09:04.074993 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.074946    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8001/health\": dial tcp 10.132.0.38:8001: connect: connection refused"
Sep 16 19:09:04.305046 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.305008    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="llm-d-routing-sidecar" containerID="cri-o://f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1" gracePeriod=60
Sep 16 19:09:04.561256 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.561230    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:09:04.606292 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606254    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-home\") pod \"40395d54-188c-421d-8d22-9d891274d3b8\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") "
Sep 16 19:09:04.606491 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40395d54-188c-421d-8d22-9d891274d3b8-tls-certs\") pod \"40395d54-188c-421d-8d22-9d891274d3b8\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") "
Sep 16 19:09:04.606491 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-model-cache\") pod \"40395d54-188c-421d-8d22-9d891274d3b8\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") "
Sep 16 19:09:04.606491 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606385    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-tmp-dir\") pod \"40395d54-188c-421d-8d22-9d891274d3b8\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") "
Sep 16 19:09:04.606491 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606423    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gktj6\" (UniqueName: \"kubernetes.io/projected/40395d54-188c-421d-8d22-9d891274d3b8-kube-api-access-gktj6\") pod \"40395d54-188c-421d-8d22-9d891274d3b8\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") "
Sep 16 19:09:04.606767 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606494    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-dshm\") pod \"40395d54-188c-421d-8d22-9d891274d3b8\" (UID: \"40395d54-188c-421d-8d22-9d891274d3b8\") "
Sep 16 19:09:04.606767 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606629    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-home" (OuterVolumeSpecName: "home") pod "40395d54-188c-421d-8d22-9d891274d3b8" (UID: "40395d54-188c-421d-8d22-9d891274d3b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:04.606767 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606651    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "40395d54-188c-421d-8d22-9d891274d3b8" (UID: "40395d54-188c-421d-8d22-9d891274d3b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:04.606767 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606720    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40395d54-188c-421d-8d22-9d891274d3b8" (UID: "40395d54-188c-421d-8d22-9d891274d3b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:04.606971 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606849    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.606971 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606869    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.606971 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.606907    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.608698 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.608669    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-dshm" (OuterVolumeSpecName: "dshm") pod "40395d54-188c-421d-8d22-9d891274d3b8" (UID: "40395d54-188c-421d-8d22-9d891274d3b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:04.608698 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.608693    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40395d54-188c-421d-8d22-9d891274d3b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40395d54-188c-421d-8d22-9d891274d3b8" (UID: "40395d54-188c-421d-8d22-9d891274d3b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:04.608855 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.608755    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40395d54-188c-421d-8d22-9d891274d3b8-kube-api-access-gktj6" (OuterVolumeSpecName: "kube-api-access-gktj6") pod "40395d54-188c-421d-8d22-9d891274d3b8" (UID: "40395d54-188c-421d-8d22-9d891274d3b8"). InnerVolumeSpecName "kube-api-access-gktj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:04.707557 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.707461    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40395d54-188c-421d-8d22-9d891274d3b8-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.707557 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.707493    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40395d54-188c-421d-8d22-9d891274d3b8-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.707557 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.707509    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gktj6\" (UniqueName: \"kubernetes.io/projected/40395d54-188c-421d-8d22-9d891274d3b8-kube-api-access-gktj6\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:09:04.787421 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.787385    2561 generic.go:356] "Generic (PLEG): container finished" podID="40395d54-188c-421d-8d22-9d891274d3b8" containerID="7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208" exitCode=0
Sep 16 19:09:04.787421 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.787414    2561 generic.go:356] "Generic (PLEG): container finished" podID="40395d54-188c-421d-8d22-9d891274d3b8" containerID="f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1" exitCode=0
Sep 16 19:09:04.787667 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.787439    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"
Sep 16 19:09:04.787667 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.787440    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" event={"ID":"40395d54-188c-421d-8d22-9d891274d3b8","Type":"ContainerDied","Data":"7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208"}
Sep 16 19:09:04.787667 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.787504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" event={"ID":"40395d54-188c-421d-8d22-9d891274d3b8","Type":"ContainerDied","Data":"f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1"}
Sep 16 19:09:04.787667 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.787517    2561 scope.go:117] "RemoveContainer" containerID="7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208"
Sep 16 19:09:04.787667 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.787524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs" event={"ID":"40395d54-188c-421d-8d22-9d891274d3b8","Type":"ContainerDied","Data":"59d5e5c0e0f42fdaa2a34e61df5fd83684c5624aeb5f907f343cccd2e0ffdd3e"}
Sep 16 19:09:04.795958 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.795939    2561 scope.go:117] "RemoveContainer" containerID="f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1"
Sep 16 19:09:04.804491 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.804475    2561 scope.go:117] "RemoveContainer" containerID="7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208"
Sep 16 19:09:04.804770 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:09:04.804750    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208\": container with ID starting with 7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208 not found: ID does not exist" containerID="7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208"
Sep 16 19:09:04.804841 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.804778    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208"} err="failed to get container status \"7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208\": rpc error: code = NotFound desc = could not find container \"7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208\": container with ID starting with 7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208 not found: ID does not exist"
Sep 16 19:09:04.804841 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.804795    2561 scope.go:117] "RemoveContainer" containerID="f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1"
Sep 16 19:09:04.805067 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:09:04.805052    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1\": container with ID starting with f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1 not found: ID does not exist" containerID="f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1"
Sep 16 19:09:04.805122 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.805069    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1"} err="failed to get container status \"f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1\": rpc error: code = NotFound desc = could not find container \"f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1\": container with ID starting with f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1 not found: ID does not exist"
Sep 16 19:09:04.805122 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.805081    2561 scope.go:117] "RemoveContainer" containerID="7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208"
Sep 16 19:09:04.805306 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.805291    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208"} err="failed to get container status \"7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208\": rpc error: code = NotFound desc = could not find container \"7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208\": container with ID starting with 7e35ff52c6b04e36e607851046873fb203b26a8f97162a19fd1d03a822cd8208 not found: ID does not exist"
Sep 16 19:09:04.805352 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.805306    2561 scope.go:117] "RemoveContainer" containerID="f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1"
Sep 16 19:09:04.805546 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.805526    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1"} err="failed to get container status \"f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1\": rpc error: code = NotFound desc = could not find container \"f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1\": container with ID starting with f75a0af5b3a6a19ec650b9d0ae1502ded23cbe7ef20c287c720b0a7637b7f1c1 not found: ID does not exist"
Sep 16 19:09:04.820264 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.820237    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"]
Sep 16 19:09:04.828248 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:04.828223    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5mcfs"]
Sep 16 19:09:06.749370 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:06.749331    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40395d54-188c-421d-8d22-9d891274d3b8" path="/var/lib/kubelet/pods/40395d54-188c-421d-8d22-9d891274d3b8/volumes"
Sep 16 19:09:18.786895 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:18.786864    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:09:18.787864 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:18.787838    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:09:34.396916 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.396884    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"]
Sep 16 19:09:34.397362 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.397209    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="llm-d-routing-sidecar"
Sep 16 19:09:34.397362 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.397221    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="llm-d-routing-sidecar"
Sep 16 19:09:34.397362 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.397229    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="main"
Sep 16 19:09:34.397362 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.397234    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="main"
Sep 16 19:09:34.397362 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.397295    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="llm-d-routing-sidecar"
Sep 16 19:09:34.397362 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.397303    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40395d54-188c-421d-8d22-9d891274d3b8" containerName="main"
Sep 16 19:09:34.401361 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.401340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.411105 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.411080    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-dsvxd\""
Sep 16 19:09:34.412013 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.411988    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:09:34.412135 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.412016    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 16 19:09:34.412135 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.412069    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:09:34.424287 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.424255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"]
Sep 16 19:09:34.461980 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.461947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-dshm\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.462157 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.462001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.462157 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.462026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tls-certs\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.462157 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.462084    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-home\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.462157 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.462115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrlb\" (UniqueName: \"kubernetes.io/projected/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-kube-api-access-fsrlb\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.462157 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.462137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-model-cache\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.563252 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.563209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.563252 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.563255    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tls-certs\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.563522 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.563303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-home\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.563522 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.563325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrlb\" (UniqueName: \"kubernetes.io/projected/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-kube-api-access-fsrlb\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.563522 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.563350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-model-cache\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.563522 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.563385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-dshm\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.563781 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.563712    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.563781 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.563749    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-home\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.563874 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.563850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-model-cache\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.565796 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.565769    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-dshm\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.565947 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.565932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tls-certs\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.575269 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.575243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrlb\" (UniqueName: \"kubernetes.io/projected/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-kube-api-access-fsrlb\") pod \"stop-v1-kserve-5657b88c95-v956j\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:34.712601 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:34.712478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:35.002787 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:35.002682    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"]
Sep 16 19:09:35.005920 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:09:35.005896    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa3da20_3e8a_4f90_ad0c_77c3dd8582b6.slice/crio-cd8a846aba5d400e6d159200ae2c07211931cfacc7ff55ca2f74b533f5b6a8fe WatchSource:0}: Error finding container cd8a846aba5d400e6d159200ae2c07211931cfacc7ff55ca2f74b533f5b6a8fe: Status 404 returned error can't find the container with id cd8a846aba5d400e6d159200ae2c07211931cfacc7ff55ca2f74b533f5b6a8fe
Sep 16 19:09:35.923813 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:35.923780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j" event={"ID":"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6","Type":"ContainerStarted","Data":"cd8a846aba5d400e6d159200ae2c07211931cfacc7ff55ca2f74b533f5b6a8fe"}
Sep 16 19:09:36.929978 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:36.929939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j" event={"ID":"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6","Type":"ContainerStarted","Data":"41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7"}
Sep 16 19:09:37.713378 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:37.713337    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:44.713406 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:44.713357    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:44.726032 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:44.726006    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:44.749072 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:44.749013    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j" podStartSLOduration=9.759204371 podStartE2EDuration="10.748991185s" totalImagesPullingTime="989.786814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:36.958445933 +0000 UTC m=+1818.746774299" watchObservedRunningTime="2026-09-16 19:09:44.748991185 +0000 UTC m=+1826.537319551"
Sep 16 19:09:44.978865 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:44.978835    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:09:59.929587 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.929528    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"]
Sep 16 19:09:59.933048 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.933024    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:09:59.936889 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.936854    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:09:59.936889 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.936851    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-6rp7b\""
Sep 16 19:09:59.937104 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.936920    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:09:59.937104 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.936937    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:09:59.937225 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.937196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-4kp45\""
Sep 16 19:09:59.950552 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.950522    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"]
Sep 16 19:09:59.997913 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.997877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:09:59.997913 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.997914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:09:59.998171 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.997985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee96d70-c92c-47b1-8e2a-79210055905a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:09:59.998171 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.998002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzrx6\" (UniqueName: \"kubernetes.io/projected/1ee96d70-c92c-47b1-8e2a-79210055905a-kube-api-access-fzrx6\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:09:59.998171 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.998036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:09:59.998171 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:09:59.998125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.099371 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.099332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.099371 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.099368    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.099655 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.099414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee96d70-c92c-47b1-8e2a-79210055905a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.099655 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.099434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzrx6\" (UniqueName: \"kubernetes.io/projected/1ee96d70-c92c-47b1-8e2a-79210055905a-kube-api-access-fzrx6\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.099741 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.099708    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.099797 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.099779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.099843 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.099785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.099843 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.099829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.100120 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.100101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.101811 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.101790    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.102122 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.102102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee96d70-c92c-47b1-8e2a-79210055905a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.108218 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.108193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzrx6\" (UniqueName: \"kubernetes.io/projected/1ee96d70-c92c-47b1-8e2a-79210055905a-kube-api-access-fzrx6\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.242762 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.242722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:00.521207 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:00.521177    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"]
Sep 16 19:10:00.522659 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:10:00.522627    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee96d70_c92c_47b1_8e2a_79210055905a.slice/crio-7a8b399d59ea65b3a60e98079758bcd1c91eb867a7848e17562c5660f8a7cdbd WatchSource:0}: Error finding container 7a8b399d59ea65b3a60e98079758bcd1c91eb867a7848e17562c5660f8a7cdbd: Status 404 returned error can't find the container with id 7a8b399d59ea65b3a60e98079758bcd1c91eb867a7848e17562c5660f8a7cdbd
Sep 16 19:10:01.039070 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:01.039035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" event={"ID":"1ee96d70-c92c-47b1-8e2a-79210055905a","Type":"ContainerStarted","Data":"8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c"}
Sep 16 19:10:01.039478 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:01.039086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" event={"ID":"1ee96d70-c92c-47b1-8e2a-79210055905a","Type":"ContainerStarted","Data":"7a8b399d59ea65b3a60e98079758bcd1c91eb867a7848e17562c5660f8a7cdbd"}
Sep 16 19:10:01.039478 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:01.039135    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:02.045869 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:02.045830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" event={"ID":"1ee96d70-c92c-47b1-8e2a-79210055905a","Type":"ContainerStarted","Data":"cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e"}
Sep 16 19:10:02.070690 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:02.070631    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" podStartSLOduration=3.070612076 podStartE2EDuration="3.070612076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:02.068691668 +0000 UTC m=+1843.857020061" watchObservedRunningTime="2026-09-16 19:10:02.070612076 +0000 UTC m=+1843.858940442"
Sep 16 19:10:02.243164 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:02.243112    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:10.243292 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:10.243246    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:10.256381 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:10.256350    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:10.261545 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:10.261517    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:11.098170 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:11.098133    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:43.737100 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:43.737013    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"]
Sep 16 19:10:43.746119 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:10:43.746078    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:10:43.746119 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:10:43.746119 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:10:43.746119 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e" cmd=["/bin/sleep","15"]
Sep 16 19:10:43.746335 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:43.746143    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" message="" err=<
Sep 16 19:10:43.746335 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:10:43.746335 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:10:43.746335 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 19:10:43.746335 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:10:43.746159    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:10:43.746335 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:10:43.746335 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:10:43.746335 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="main" containerID="cri-o://cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e"
Sep 16 19:10:43.746335 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:43.746206    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="main" containerID="cri-o://cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e" gracePeriod=60
Sep 16 19:10:44.087883 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.087785    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8001/health\": dial tcp 10.132.0.40:8001: connect: connection refused"
Sep 16 19:10:44.239704 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.239667    2561 generic.go:356] "Generic (PLEG): container finished" podID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerID="cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e" exitCode=0
Sep 16 19:10:44.239885 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.239736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" event={"ID":"1ee96d70-c92c-47b1-8e2a-79210055905a","Type":"ContainerDied","Data":"cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e"}
Sep 16 19:10:44.359454 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.359423    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="llm-d-routing-sidecar" containerID="cri-o://8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c" gracePeriod=60
Sep 16 19:10:44.608263 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.608236    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:44.703055 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.702959    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-model-cache\") pod \"1ee96d70-c92c-47b1-8e2a-79210055905a\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") "
Sep 16 19:10:44.703055 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703008    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-home\") pod \"1ee96d70-c92c-47b1-8e2a-79210055905a\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") "
Sep 16 19:10:44.703055 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703027    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-dshm\") pod \"1ee96d70-c92c-47b1-8e2a-79210055905a\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") "
Sep 16 19:10:44.703372 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703061    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzrx6\" (UniqueName: \"kubernetes.io/projected/1ee96d70-c92c-47b1-8e2a-79210055905a-kube-api-access-fzrx6\") pod \"1ee96d70-c92c-47b1-8e2a-79210055905a\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") "
Sep 16 19:10:44.703372 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703123    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee96d70-c92c-47b1-8e2a-79210055905a-tls-certs\") pod \"1ee96d70-c92c-47b1-8e2a-79210055905a\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") "
Sep 16 19:10:44.703372 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703176    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-tmp-dir\") pod \"1ee96d70-c92c-47b1-8e2a-79210055905a\" (UID: \"1ee96d70-c92c-47b1-8e2a-79210055905a\") "
Sep 16 19:10:44.703372 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-home" (OuterVolumeSpecName: "home") pod "1ee96d70-c92c-47b1-8e2a-79210055905a" (UID: "1ee96d70-c92c-47b1-8e2a-79210055905a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:44.703372 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703304    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ee96d70-c92c-47b1-8e2a-79210055905a" (UID: "1ee96d70-c92c-47b1-8e2a-79210055905a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:44.703655 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703460    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.703655 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703475    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.703655 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.703497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ee96d70-c92c-47b1-8e2a-79210055905a" (UID: "1ee96d70-c92c-47b1-8e2a-79210055905a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:44.705425 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.705397    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-dshm" (OuterVolumeSpecName: "dshm") pod "1ee96d70-c92c-47b1-8e2a-79210055905a" (UID: "1ee96d70-c92c-47b1-8e2a-79210055905a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:44.705552 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.705473    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee96d70-c92c-47b1-8e2a-79210055905a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ee96d70-c92c-47b1-8e2a-79210055905a" (UID: "1ee96d70-c92c-47b1-8e2a-79210055905a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:44.705552 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.705489    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee96d70-c92c-47b1-8e2a-79210055905a-kube-api-access-fzrx6" (OuterVolumeSpecName: "kube-api-access-fzrx6") pod "1ee96d70-c92c-47b1-8e2a-79210055905a" (UID: "1ee96d70-c92c-47b1-8e2a-79210055905a"). InnerVolumeSpecName "kube-api-access-fzrx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:44.804593 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.804537    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee96d70-c92c-47b1-8e2a-79210055905a-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.804593 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.804590    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.804989 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.804601    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee96d70-c92c-47b1-8e2a-79210055905a-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:10:44.804989 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:44.804612    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzrx6\" (UniqueName: \"kubernetes.io/projected/1ee96d70-c92c-47b1-8e2a-79210055905a-kube-api-access-fzrx6\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:10:45.246019 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.245984    2561 generic.go:356] "Generic (PLEG): container finished" podID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerID="8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c" exitCode=0
Sep 16 19:10:45.246228 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.246031    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"
Sep 16 19:10:45.246228 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.246068    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" event={"ID":"1ee96d70-c92c-47b1-8e2a-79210055905a","Type":"ContainerDied","Data":"8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c"}
Sep 16 19:10:45.246228 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.246130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk" event={"ID":"1ee96d70-c92c-47b1-8e2a-79210055905a","Type":"ContainerDied","Data":"7a8b399d59ea65b3a60e98079758bcd1c91eb867a7848e17562c5660f8a7cdbd"}
Sep 16 19:10:45.246228 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.246163    2561 scope.go:117] "RemoveContainer" containerID="cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e"
Sep 16 19:10:45.254998 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.254977    2561 scope.go:117] "RemoveContainer" containerID="8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c"
Sep 16 19:10:45.262754 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.262735    2561 scope.go:117] "RemoveContainer" containerID="cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e"
Sep 16 19:10:45.263048 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:10:45.263027    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e\": container with ID starting with cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e not found: ID does not exist" containerID="cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e"
Sep 16 19:10:45.263144 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.263060    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e"} err="failed to get container status \"cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e\": rpc error: code = NotFound desc = could not find container \"cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e\": container with ID starting with cd3a90d72cc580a707583023060c114ea43743ba4d98030fa4c59fddf1764a2e not found: ID does not exist"
Sep 16 19:10:45.263144 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.263089    2561 scope.go:117] "RemoveContainer" containerID="8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c"
Sep 16 19:10:45.263368 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:10:45.263351    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c\": container with ID starting with 8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c not found: ID does not exist" containerID="8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c"
Sep 16 19:10:45.263414 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.263383    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c"} err="failed to get container status \"8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c\": rpc error: code = NotFound desc = could not find container \"8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c\": container with ID starting with 8808c611d582a0066c2f4fa4e898861deb396f782da14d8b235b161e7a5ef16c not found: ID does not exist"
Sep 16 19:10:45.278793 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.278752    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"]
Sep 16 19:10:45.280223 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:45.280183    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-znmhk"]
Sep 16 19:10:46.755082 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:46.755033    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" path="/var/lib/kubelet/pods/1ee96d70-c92c-47b1-8e2a-79210055905a/volumes"
Sep 16 19:10:53.916817 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:10:53.916782    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"]
Sep 16 19:11:08.935691 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:08.935645    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j" podUID="2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" containerName="main" containerID="cri-o://41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7" gracePeriod=45
Sep 16 19:11:08.968487 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:08.968440    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j" podUID="2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 19:11:09.174278 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.174248    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:11:09.218679 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.218588    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-dshm\") pod \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") "
Sep 16 19:11:09.218679 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.218624    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-home\") pod \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") "
Sep 16 19:11:09.218679 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.218672    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-model-cache\") pod \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") "
Sep 16 19:11:09.218963 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.218691    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsrlb\" (UniqueName: \"kubernetes.io/projected/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-kube-api-access-fsrlb\") pod \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") "
Sep 16 19:11:09.218963 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.218712    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tmp-dir\") pod \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") "
Sep 16 19:11:09.218963 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.218734    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tls-certs\") pod \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\" (UID: \"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6\") "
Sep 16 19:11:09.218963 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.218935    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-home" (OuterVolumeSpecName: "home") pod "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" (UID: "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:09.219169 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.218972    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" (UID: "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:09.219169 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.219012    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" (UID: "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:09.219169 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.219024    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.220992 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.220966    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-kube-api-access-fsrlb" (OuterVolumeSpecName: "kube-api-access-fsrlb") pod "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" (UID: "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6"). InnerVolumeSpecName "kube-api-access-fsrlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:09.221085 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.220995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-dshm" (OuterVolumeSpecName: "dshm") pod "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" (UID: "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:09.221085 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.221030    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" (UID: "2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:09.319696 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.319662    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.319696 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.319691    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.319696 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.319702    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsrlb\" (UniqueName: \"kubernetes.io/projected/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-kube-api-access-fsrlb\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.319931 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.319711    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.319931 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.319722    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:11:09.360503 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.360460    2561 generic.go:356] "Generic (PLEG): container finished" podID="2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" containerID="41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7" exitCode=0
Sep 16 19:11:09.360714 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.360510    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"
Sep 16 19:11:09.360714 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.360530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j" event={"ID":"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6","Type":"ContainerDied","Data":"41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7"}
Sep 16 19:11:09.360714 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.360610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j" event={"ID":"2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6","Type":"ContainerDied","Data":"cd8a846aba5d400e6d159200ae2c07211931cfacc7ff55ca2f74b533f5b6a8fe"}
Sep 16 19:11:09.360714 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.360641    2561 scope.go:117] "RemoveContainer" containerID="41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7"
Sep 16 19:11:09.369275 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.369252    2561 scope.go:117] "RemoveContainer" containerID="41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7"
Sep 16 19:11:09.369547 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:11:09.369529    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7\": container with ID starting with 41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7 not found: ID does not exist" containerID="41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7"
Sep 16 19:11:09.369624 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.369558    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7"} err="failed to get container status \"41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7\": rpc error: code = NotFound desc = could not find container \"41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7\": container with ID starting with 41b1ae5daf2e359770f1bb4882429178cf06e23bdf0ec411f7ccd8d6eb2aebe7 not found: ID does not exist"
Sep 16 19:11:09.383206 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.383182    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"]
Sep 16 19:11:09.387473 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:09.387450    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-v956j"]
Sep 16 19:11:10.748941 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:10.748907    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" path="/var/lib/kubelet/pods/2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6/volumes"
Sep 16 19:11:15.070538 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070505    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"]
Sep 16 19:11:15.070924 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070854    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" containerName="main"
Sep 16 19:11:15.070924 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070867    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" containerName="main"
Sep 16 19:11:15.070924 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070876    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="llm-d-routing-sidecar"
Sep 16 19:11:15.070924 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070881    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="llm-d-routing-sidecar"
Sep 16 19:11:15.070924 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070892    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="main"
Sep 16 19:11:15.070924 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070898    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="main"
Sep 16 19:11:15.071130 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070960    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aa3da20-3e8a-4f90-ad0c-77c3dd8582b6" containerName="main"
Sep 16 19:11:15.071130 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070968    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="main"
Sep 16 19:11:15.071130 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.070977    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee96d70-c92c-47b1-8e2a-79210055905a" containerName="llm-d-routing-sidecar"
Sep 16 19:11:15.075317 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.075296    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.079507 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.079485    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 16 19:11:15.079651 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.079531    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 19:11:15.079922 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.079908    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 19:11:15.080102 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.080089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-tp2mk\""
Sep 16 19:11:15.090524 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.090497    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"]
Sep 16 19:11:15.171605 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.171544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-dshm\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.171605 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.171604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-model-cache\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.171851 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.171679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-tmp-dir\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.171851 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.171733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbng\" (UniqueName: \"kubernetes.io/projected/b337970a-a373-4900-b073-5ef615406be1-kube-api-access-mrbng\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.171851 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.171800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-home\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.171851 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.171836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b337970a-a373-4900-b073-5ef615406be1-tls-certs\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.272879 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.272835    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-dshm\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.272879 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.272873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-model-cache\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.273139 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.272897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-tmp-dir\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.273139 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.272921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbng\" (UniqueName: \"kubernetes.io/projected/b337970a-a373-4900-b073-5ef615406be1-kube-api-access-mrbng\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.273139 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.272961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-home\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.273139 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.272997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b337970a-a373-4900-b073-5ef615406be1-tls-certs\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.273371 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.273344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-tmp-dir\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.273425 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.273353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-model-cache\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.273425 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.273406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-home\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.275220 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.275188    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-dshm\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.275521 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.275500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b337970a-a373-4900-b073-5ef615406be1-tls-certs\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.283958 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.283933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbng\" (UniqueName: \"kubernetes.io/projected/b337970a-a373-4900-b073-5ef615406be1-kube-api-access-mrbng\") pod \"decom-v2-kserve-77f867d787-525s8\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.384901 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.384796    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:15.662514 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:15.662487    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"]
Sep 16 19:11:15.664517 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:11:15.664488    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb337970a_a373_4900_b073_5ef615406be1.slice/crio-cfb4a218f55a457c4dd0d9dcdd19edcc0227059930ff668f9dee4e0377d6f00d WatchSource:0}: Error finding container cfb4a218f55a457c4dd0d9dcdd19edcc0227059930ff668f9dee4e0377d6f00d: Status 404 returned error can't find the container with id cfb4a218f55a457c4dd0d9dcdd19edcc0227059930ff668f9dee4e0377d6f00d
Sep 16 19:11:16.394893 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:16.394858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8" event={"ID":"b337970a-a373-4900-b073-5ef615406be1","Type":"ContainerStarted","Data":"1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9"}
Sep 16 19:11:16.395298 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:16.394909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8" event={"ID":"b337970a-a373-4900-b073-5ef615406be1","Type":"ContainerStarted","Data":"cfb4a218f55a457c4dd0d9dcdd19edcc0227059930ff668f9dee4e0377d6f00d"}
Sep 16 19:11:16.420251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:16.420199    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8" podStartSLOduration=1.420183281 podStartE2EDuration="1.420183281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:16.415429547 +0000 UTC m=+1918.203757913" watchObservedRunningTime="2026-09-16 19:11:16.420183281 +0000 UTC m=+1918.208511646"
Sep 16 19:11:17.385655 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:17.385617    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:25.385978 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:25.385928    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:25.398910 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:25.398874    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:25.451813 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:25.451784    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:11:38.605435 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.605392    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"]
Sep 16 19:11:38.610220 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.610203    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.612945 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.612922    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:11:38.614069 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.614046    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:11:38.614288 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.614268    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:11:38.614796 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.614777    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-frc9h\""
Sep 16 19:11:38.614846 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.614789    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:11:38.615090 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.615073    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:11:38.632805 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.632776    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"]
Sep 16 19:11:38.783837 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.783793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.783837 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.783838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-dshm\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.784053 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.783867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.784053 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.783939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-model-cache\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.784053 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.784003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.784053 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.784020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdw5x\" (UniqueName: \"kubernetes.io/projected/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kube-api-access-zdw5x\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.784189 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.784056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.784189 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.784110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-home\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.885063 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.884970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.885063 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-dshm\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.885063 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.885334 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-model-cache\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.885334 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.885334 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdw5x\" (UniqueName: \"kubernetes.io/projected/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kube-api-access-zdw5x\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.885502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.885559 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885507    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-model-cache\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.885559 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-home\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.886076 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-home\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.886076 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.886076 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.885877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.886267 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.886174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.887491 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.887476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-dshm\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.887997 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.887982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.901860 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.901834    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdw5x\" (UniqueName: \"kubernetes.io/projected/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kube-api-access-zdw5x\") pod \"tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.919898 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.919866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:38.953031 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.952999    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"]
Sep 16 19:11:38.956477 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.956459    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:11:38.966631 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.966607    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-djr9n\""
Sep 16 19:11:38.979091 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:38.979063    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"]
Sep 16 19:11:39.087161 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.087125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrwq\" (UniqueName: \"kubernetes.io/projected/4a37a319-dfcf-4298-a722-b1bec0c12941-kube-api-access-xwrwq\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7\" (UID: \"4a37a319-dfcf-4298-a722-b1bec0c12941\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:11:39.087365 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.087272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a37a319-dfcf-4298-a722-b1bec0c12941-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7\" (UID: \"4a37a319-dfcf-4298-a722-b1bec0c12941\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:11:39.188313 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.188224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a37a319-dfcf-4298-a722-b1bec0c12941-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7\" (UID: \"4a37a319-dfcf-4298-a722-b1bec0c12941\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:11:39.188313 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.188298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrwq\" (UniqueName: \"kubernetes.io/projected/4a37a319-dfcf-4298-a722-b1bec0c12941-kube-api-access-xwrwq\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7\" (UID: \"4a37a319-dfcf-4298-a722-b1bec0c12941\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:11:39.190804 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.190777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a37a319-dfcf-4298-a722-b1bec0c12941-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7\" (UID: \"4a37a319-dfcf-4298-a722-b1bec0c12941\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:11:39.203817 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.203792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrwq\" (UniqueName: \"kubernetes.io/projected/4a37a319-dfcf-4298-a722-b1bec0c12941-kube-api-access-xwrwq\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7\" (UID: \"4a37a319-dfcf-4298-a722-b1bec0c12941\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:11:39.210081 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.210059    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"]
Sep 16 19:11:39.212779 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:11:39.212745    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1ba9b2_6e2c_4a12_99a7_8f7eda4bbfc2.slice/crio-e46a90897e61cf6e8c1ab46d88d2e2e43f59850c3e0e5cddae253daa99575988 WatchSource:0}: Error finding container e46a90897e61cf6e8c1ab46d88d2e2e43f59850c3e0e5cddae253daa99575988: Status 404 returned error can't find the container with id e46a90897e61cf6e8c1ab46d88d2e2e43f59850c3e0e5cddae253daa99575988
Sep 16 19:11:39.268344 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.268318    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:11:39.507959 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.507919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" event={"ID":"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2","Type":"ContainerStarted","Data":"c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c"}
Sep 16 19:11:39.508127 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.508000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" event={"ID":"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2","Type":"ContainerStarted","Data":"e46a90897e61cf6e8c1ab46d88d2e2e43f59850c3e0e5cddae253daa99575988"}
Sep 16 19:11:39.560453 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:11:39.560421    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a37a319_dfcf_4298_a722_b1bec0c12941.slice/crio-e19ee3a7638358035eb28a4ec259d05469b26323c3052a89b66cd09db3926fac WatchSource:0}: Error finding container e19ee3a7638358035eb28a4ec259d05469b26323c3052a89b66cd09db3926fac: Status 404 returned error can't find the container with id e19ee3a7638358035eb28a4ec259d05469b26323c3052a89b66cd09db3926fac
Sep 16 19:11:39.561250 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:39.561222    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"]
Sep 16 19:11:40.513730 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:40.513693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" event={"ID":"4a37a319-dfcf-4298-a722-b1bec0c12941","Type":"ContainerStarted","Data":"651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3"}
Sep 16 19:11:40.514194 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:40.513750    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" event={"ID":"4a37a319-dfcf-4298-a722-b1bec0c12941","Type":"ContainerStarted","Data":"e19ee3a7638358035eb28a4ec259d05469b26323c3052a89b66cd09db3926fac"}
Sep 16 19:11:40.514194 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:40.513804    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:11:40.550637 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:40.550548    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" podStartSLOduration=2.55052497 podStartE2EDuration="2.55052497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:40.549155074 +0000 UTC m=+1942.337483442" watchObservedRunningTime="2026-09-16 19:11:40.55052497 +0000 UTC m=+1942.338853338"
Sep 16 19:11:42.524109 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:42.524073    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" containerID="c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c" exitCode=0
Sep 16 19:11:42.524483 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:42.524148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" event={"ID":"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2","Type":"ContainerDied","Data":"c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c"}
Sep 16 19:11:43.530525 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:43.530489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" event={"ID":"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2","Type":"ContainerStarted","Data":"959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f"}
Sep 16 19:11:43.555788 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:43.555730    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" podStartSLOduration=3.32031342 podStartE2EDuration="5.555707037s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.235393617s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:43.555344703 +0000 UTC m=+1945.343673083" watchObservedRunningTime="2026-09-16 19:11:43.555707037 +0000 UTC m=+1945.344035406"
Sep 16 19:11:43.920662 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:43.920549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:48.920726 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:48.920685    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:48.933402 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:48.933373    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:49.569533 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:49.569496    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:11:50.894454 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:11:50.894418    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"]
Sep 16 19:12:05.913413 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:05.913370    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8" podUID="b337970a-a373-4900-b073-5ef615406be1" containerName="main" containerID="cri-o://1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9" gracePeriod=45
Sep 16 19:12:06.154130 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.154104    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:12:06.226524 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.226495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrbng\" (UniqueName: \"kubernetes.io/projected/b337970a-a373-4900-b073-5ef615406be1-kube-api-access-mrbng\") pod \"b337970a-a373-4900-b073-5ef615406be1\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") "
Sep 16 19:12:06.226702 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.226536    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b337970a-a373-4900-b073-5ef615406be1-tls-certs\") pod \"b337970a-a373-4900-b073-5ef615406be1\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") "
Sep 16 19:12:06.226702 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.226589    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-dshm\") pod \"b337970a-a373-4900-b073-5ef615406be1\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") "
Sep 16 19:12:06.226831 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.226713    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-tmp-dir\") pod \"b337970a-a373-4900-b073-5ef615406be1\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") "
Sep 16 19:12:06.226831 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.226766    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-home\") pod \"b337970a-a373-4900-b073-5ef615406be1\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") "
Sep 16 19:12:06.226931 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.226848    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-model-cache\") pod \"b337970a-a373-4900-b073-5ef615406be1\" (UID: \"b337970a-a373-4900-b073-5ef615406be1\") "
Sep 16 19:12:06.227046 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.226991    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b337970a-a373-4900-b073-5ef615406be1" (UID: "b337970a-a373-4900-b073-5ef615406be1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:06.227179 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.227068    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-home" (OuterVolumeSpecName: "home") pod "b337970a-a373-4900-b073-5ef615406be1" (UID: "b337970a-a373-4900-b073-5ef615406be1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:06.227179 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.227141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b337970a-a373-4900-b073-5ef615406be1" (UID: "b337970a-a373-4900-b073-5ef615406be1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:06.227306 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.227197    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:12:06.227306 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.227217    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:12:06.228910 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.228890    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-dshm" (OuterVolumeSpecName: "dshm") pod "b337970a-a373-4900-b073-5ef615406be1" (UID: "b337970a-a373-4900-b073-5ef615406be1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:06.228980 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.228916    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b337970a-a373-4900-b073-5ef615406be1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b337970a-a373-4900-b073-5ef615406be1" (UID: "b337970a-a373-4900-b073-5ef615406be1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:06.229021 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.229002    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b337970a-a373-4900-b073-5ef615406be1-kube-api-access-mrbng" (OuterVolumeSpecName: "kube-api-access-mrbng") pod "b337970a-a373-4900-b073-5ef615406be1" (UID: "b337970a-a373-4900-b073-5ef615406be1"). InnerVolumeSpecName "kube-api-access-mrbng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:06.327752 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.327716    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:12:06.327752 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.327744    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b337970a-a373-4900-b073-5ef615406be1-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:12:06.327752 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.327756    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrbng\" (UniqueName: \"kubernetes.io/projected/b337970a-a373-4900-b073-5ef615406be1-kube-api-access-mrbng\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:12:06.327979 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.327764    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b337970a-a373-4900-b073-5ef615406be1-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:12:06.638614 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.638505    2561 generic.go:356] "Generic (PLEG): container finished" podID="b337970a-a373-4900-b073-5ef615406be1" containerID="1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9" exitCode=0
Sep 16 19:12:06.638614 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.638548    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"
Sep 16 19:12:06.638836 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.638604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8" event={"ID":"b337970a-a373-4900-b073-5ef615406be1","Type":"ContainerDied","Data":"1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9"}
Sep 16 19:12:06.638836 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.638669    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8" event={"ID":"b337970a-a373-4900-b073-5ef615406be1","Type":"ContainerDied","Data":"cfb4a218f55a457c4dd0d9dcdd19edcc0227059930ff668f9dee4e0377d6f00d"}
Sep 16 19:12:06.638836 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.638693    2561 scope.go:117] "RemoveContainer" containerID="1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9"
Sep 16 19:12:06.648444 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.648422    2561 scope.go:117] "RemoveContainer" containerID="1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9"
Sep 16 19:12:06.648795 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:12:06.648776    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9\": container with ID starting with 1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9 not found: ID does not exist" containerID="1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9"
Sep 16 19:12:06.648863 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.648804    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9"} err="failed to get container status \"1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9\": rpc error: code = NotFound desc = could not find container \"1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9\": container with ID starting with 1952e156e287211a4079f3625c8747d3db3ac4c23a5538af7cf41cd73bcb5df9 not found: ID does not exist"
Sep 16 19:12:06.670432 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.670398    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"]
Sep 16 19:12:06.678259 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.678225    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-525s8"]
Sep 16 19:12:06.749312 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:06.749274    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b337970a-a373-4900-b073-5ef615406be1" path="/var/lib/kubelet/pods/b337970a-a373-4900-b073-5ef615406be1/volumes"
Sep 16 19:12:11.519469 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:12:11.519393    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:13:36.615765 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:36.615728    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"]
Sep 16 19:13:36.630321 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:36.630289    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"]
Sep 16 19:13:36.639499 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:13:36.639454    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:13:36.639499 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.639499 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.639499 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f" cmd=["/bin/sleep","15"]
Sep 16 19:13:36.639801 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:36.639521    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" message="" err=<
Sep 16 19:13:36.639801 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.639801 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.639801 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 19:13:36.639801 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:13:36.639537    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:13:36.639801 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.639801 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.639801 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" podUID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" containerName="main" containerID="cri-o://959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f"
Sep 16 19:13:36.639801 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:36.639626    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" podUID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" containerName="main" containerID="cri-o://959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f" gracePeriod=60
Sep 16 19:13:36.992277 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:36.992252    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:13:37.062392 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.062360    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" containerID="959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f" exitCode=0
Sep 16 19:13:37.062614 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.062419    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"
Sep 16 19:13:37.062614 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.062444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" event={"ID":"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2","Type":"ContainerDied","Data":"959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f"}
Sep 16 19:13:37.062614 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.062503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw" event={"ID":"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2","Type":"ContainerDied","Data":"e46a90897e61cf6e8c1ab46d88d2e2e43f59850c3e0e5cddae253daa99575988"}
Sep 16 19:13:37.062614 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.062535    2561 scope.go:117] "RemoveContainer" containerID="959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f"
Sep 16 19:13:37.071335 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.071312    2561 scope.go:117] "RemoveContainer" containerID="c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c"
Sep 16 19:13:37.079496 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.079468    2561 scope.go:117] "RemoveContainer" containerID="959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f"
Sep 16 19:13:37.079813 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:13:37.079794    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f\": container with ID starting with 959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f not found: ID does not exist" containerID="959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f"
Sep 16 19:13:37.079877 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.079822    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f"} err="failed to get container status \"959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f\": rpc error: code = NotFound desc = could not find container \"959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f\": container with ID starting with 959f6216d0d599c4f4df9b53cc9b0aa26a3b2f012e2b27a4ba4708aae8147e4f not found: ID does not exist"
Sep 16 19:13:37.079877 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.079839    2561 scope.go:117] "RemoveContainer" containerID="c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c"
Sep 16 19:13:37.080141 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:13:37.080125    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c\": container with ID starting with c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c not found: ID does not exist" containerID="c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c"
Sep 16 19:13:37.080198 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.080144    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c"} err="failed to get container status \"c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c\": rpc error: code = NotFound desc = could not find container \"c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c\": container with ID starting with c45ed6e25d546d6ca0105449b28264d2c3d0799a90d31580d5d7ac3d0ad8c02c not found: ID does not exist"
Sep 16 19:13:37.126999 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.126968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-dshm\") pod \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") "
Sep 16 19:13:37.127190 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127022    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tls-certs\") pod \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") "
Sep 16 19:13:37.127190 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127039    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tmp-dir\") pod \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") "
Sep 16 19:13:37.127190 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127082    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdw5x\" (UniqueName: \"kubernetes.io/projected/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kube-api-access-zdw5x\") pod \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") "
Sep 16 19:13:37.127190 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127108    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-cabundle-cert\") pod \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") "
Sep 16 19:13:37.127190 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127133    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-home\") pod \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") "
Sep 16 19:13:37.127190 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127169    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kserve-provision-location\") pod \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") "
Sep 16 19:13:37.127536 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127227    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-model-cache\") pod \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\" (UID: \"6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2\") "
Sep 16 19:13:37.127536 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127298    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" (UID: "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.127536 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127522    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.127709 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-home" (OuterVolumeSpecName: "home") pod "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" (UID: "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.127709 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" (UID: "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.127709 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.127622    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" (UID: "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:37.129423 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.129399    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-dshm" (OuterVolumeSpecName: "dshm") pod "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" (UID: "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.129423 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.129408    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" (UID: "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:37.129604 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.129480    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kube-api-access-zdw5x" (OuterVolumeSpecName: "kube-api-access-zdw5x") pod "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" (UID: "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2"). InnerVolumeSpecName "kube-api-access-zdw5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:37.154119 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.154084    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" (UID: "6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.228425 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.228394    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.228425 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.228424    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.228638 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.228435    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.228638 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.228443    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.228638 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.228452    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.228638 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.228460    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdw5x\" (UniqueName: \"kubernetes.io/projected/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-kube-api-access-zdw5x\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.228638 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.228468    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.385270 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.385240    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"]
Sep 16 19:13:37.389257 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:37.389230    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7fd7c88dbc-bcmsw"]
Sep 16 19:13:38.749071 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:38.749036    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" path="/var/lib/kubelet/pods/6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2/volumes"
Sep 16 19:13:41.520618 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:41.520519    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:51.519953 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:51.519906    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:51.637173 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:13:51.637130    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerName="main" containerID="cri-o://651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3" gracePeriod=45
Sep 16 19:14:01.520471 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:01.520422    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:14:01.520940 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:01.520562    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:14:11.519691 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:11.519652    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:14:18.816321 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:18.816292    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:14:18.818665 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:18.818637    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:14:21.520090 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:21.520047    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:14:21.890862 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:21.890830    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:14:21.920455 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:21.920420    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwrwq\" (UniqueName: \"kubernetes.io/projected/4a37a319-dfcf-4298-a722-b1bec0c12941-kube-api-access-xwrwq\") pod \"4a37a319-dfcf-4298-a722-b1bec0c12941\" (UID: \"4a37a319-dfcf-4298-a722-b1bec0c12941\") "
Sep 16 19:14:21.920661 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:21.920532    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a37a319-dfcf-4298-a722-b1bec0c12941-tls-certs\") pod \"4a37a319-dfcf-4298-a722-b1bec0c12941\" (UID: \"4a37a319-dfcf-4298-a722-b1bec0c12941\") "
Sep 16 19:14:21.922583 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:21.922547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a37a319-dfcf-4298-a722-b1bec0c12941-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a37a319-dfcf-4298-a722-b1bec0c12941" (UID: "4a37a319-dfcf-4298-a722-b1bec0c12941"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:21.922682 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:21.922606    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a37a319-dfcf-4298-a722-b1bec0c12941-kube-api-access-xwrwq" (OuterVolumeSpecName: "kube-api-access-xwrwq") pod "4a37a319-dfcf-4298-a722-b1bec0c12941" (UID: "4a37a319-dfcf-4298-a722-b1bec0c12941"). InnerVolumeSpecName "kube-api-access-xwrwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:22.022137 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.022101    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a37a319-dfcf-4298-a722-b1bec0c12941-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:14:22.022137 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.022132    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrwq\" (UniqueName: \"kubernetes.io/projected/4a37a319-dfcf-4298-a722-b1bec0c12941-kube-api-access-xwrwq\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:14:22.266168 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.266133    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerID="651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3" exitCode=0
Sep 16 19:14:22.266338 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.266175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" event={"ID":"4a37a319-dfcf-4298-a722-b1bec0c12941","Type":"ContainerDied","Data":"651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3"}
Sep 16 19:14:22.266338 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.266179    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"
Sep 16 19:14:22.266338 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.266224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7" event={"ID":"4a37a319-dfcf-4298-a722-b1bec0c12941","Type":"ContainerDied","Data":"e19ee3a7638358035eb28a4ec259d05469b26323c3052a89b66cd09db3926fac"}
Sep 16 19:14:22.266338 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.266247    2561 scope.go:117] "RemoveContainer" containerID="651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3"
Sep 16 19:14:22.274954 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.274934    2561 scope.go:117] "RemoveContainer" containerID="651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3"
Sep 16 19:14:22.275227 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:14:22.275207    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3\": container with ID starting with 651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3 not found: ID does not exist" containerID="651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3"
Sep 16 19:14:22.275293 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.275234    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3"} err="failed to get container status \"651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3\": rpc error: code = NotFound desc = could not find container \"651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3\": container with ID starting with 651727d43552e96f690cccec779df6a9af417e5b740e80e0989739d4090c54b3 not found: ID does not exist"
Sep 16 19:14:22.290423 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.290362    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"]
Sep 16 19:14:22.292060 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.292034    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b696lmg7"]
Sep 16 19:14:22.750585 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:22.750540    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" path="/var/lib/kubelet/pods/4a37a319-dfcf-4298-a722-b1bec0c12941/volumes"
Sep 16 19:14:51.290098 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290059    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"]
Sep 16 19:14:51.290502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290423    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" containerName="main"
Sep 16 19:14:51.290502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290437    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" containerName="main"
Sep 16 19:14:51.290502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290453    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" containerName="storage-initializer"
Sep 16 19:14:51.290502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290458    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" containerName="storage-initializer"
Sep 16 19:14:51.290502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290467    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b337970a-a373-4900-b073-5ef615406be1" containerName="main"
Sep 16 19:14:51.290502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290474    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b337970a-a373-4900-b073-5ef615406be1" containerName="main"
Sep 16 19:14:51.290502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290481    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerName="main"
Sep 16 19:14:51.290502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290486    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerName="main"
Sep 16 19:14:51.290785 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290541    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d1ba9b2-6e2c-4a12-99a7-8f7eda4bbfc2" containerName="main"
Sep 16 19:14:51.290785 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290548    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a37a319-dfcf-4298-a722-b1bec0c12941" containerName="main"
Sep 16 19:14:51.290785 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.290557    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b337970a-a373-4900-b073-5ef615406be1" containerName="main"
Sep 16 19:14:51.295231 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.295202    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.300597 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.300550    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:51.300597 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.300589    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:14:51.300777 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.300563    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-gn7n2\""
Sep 16 19:14:51.300777 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.300559    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:51.300777 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.300550    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:14:51.300777 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.300596    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:14:51.316417 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.316389    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"]
Sep 16 19:14:51.369295 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.369258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1badcca0-41a4-424d-ba51-9149ebe4fe2a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.369295 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.369301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-dshm\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.369528 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.369404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqp2\" (UniqueName: \"kubernetes.io/projected/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kube-api-access-hmqp2\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.369528 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.369436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.369528 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.369459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-home\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.369528 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.369479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.369528 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.369519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.369779 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.369556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-model-cache\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.470625 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.470555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1badcca0-41a4-424d-ba51-9149ebe4fe2a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.470830 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.470635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-dshm\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.470830 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.470691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqp2\" (UniqueName: \"kubernetes.io/projected/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kube-api-access-hmqp2\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.470830 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.470709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.471039 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.470821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-home\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.471039 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.470861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.471039 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.470885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.471039 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.470911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-model-cache\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.471307 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.471196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.471307 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.471252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-home\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.471307 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.471287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-model-cache\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.471480 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.471339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.471539 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.471521    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1badcca0-41a4-424d-ba51-9149ebe4fe2a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.473080 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.473062    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-dshm\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.473350 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.473335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.482149 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.482127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqp2\" (UniqueName: \"kubernetes.io/projected/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kube-api-access-hmqp2\") pod \"tokenizer-migration-path-test-kserve-668bcccbb-hszdv\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.569880 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.569800    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"]
Sep 16 19:14:51.574485 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.574465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:14:51.577138 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.577118    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-vdj5v\""
Sep 16 19:14:51.587619 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.587595    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"]
Sep 16 19:14:51.605662 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.605634    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:14:51.672257 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.672221    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88l5t\" (UniqueName: \"kubernetes.io/projected/8a53c5e3-950f-4143-aa99-895524dc53bd-kube-api-access-88l5t\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj\" (UID: \"8a53c5e3-950f-4143-aa99-895524dc53bd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:14:51.672409 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.672279    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a53c5e3-950f-4143-aa99-895524dc53bd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj\" (UID: \"8a53c5e3-950f-4143-aa99-895524dc53bd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:14:51.773255 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.773208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88l5t\" (UniqueName: \"kubernetes.io/projected/8a53c5e3-950f-4143-aa99-895524dc53bd-kube-api-access-88l5t\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj\" (UID: \"8a53c5e3-950f-4143-aa99-895524dc53bd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:14:51.773255 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.773258    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a53c5e3-950f-4143-aa99-895524dc53bd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj\" (UID: \"8a53c5e3-950f-4143-aa99-895524dc53bd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:14:51.775801 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.775781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a53c5e3-950f-4143-aa99-895524dc53bd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj\" (UID: \"8a53c5e3-950f-4143-aa99-895524dc53bd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:14:51.790305 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.790277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88l5t\" (UniqueName: \"kubernetes.io/projected/8a53c5e3-950f-4143-aa99-895524dc53bd-kube-api-access-88l5t\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj\" (UID: \"8a53c5e3-950f-4143-aa99-895524dc53bd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:14:51.885723 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.885631    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:14:51.891900 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:51.891869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"]
Sep 16 19:14:51.892001 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:14:51.891866    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1badcca0_41a4_424d_ba51_9149ebe4fe2a.slice/crio-5ed38761b3da8cdba4d07e6753f6ec45312a33be84843d37136e5fbb6ae6e955 WatchSource:0}: Error finding container 5ed38761b3da8cdba4d07e6753f6ec45312a33be84843d37136e5fbb6ae6e955: Status 404 returned error can't find the container with id 5ed38761b3da8cdba4d07e6753f6ec45312a33be84843d37136e5fbb6ae6e955
Sep 16 19:14:52.169865 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:52.169814    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"]
Sep 16 19:14:52.172478 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:14:52.172444    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a53c5e3_950f_4143_aa99_895524dc53bd.slice/crio-aa3aa5d4c7bc2f6d3a1edfc7224090e183009997df8fa06ee628e62b6d327d94 WatchSource:0}: Error finding container aa3aa5d4c7bc2f6d3a1edfc7224090e183009997df8fa06ee628e62b6d327d94: Status 404 returned error can't find the container with id aa3aa5d4c7bc2f6d3a1edfc7224090e183009997df8fa06ee628e62b6d327d94
Sep 16 19:14:52.404762 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:52.404720    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" event={"ID":"1badcca0-41a4-424d-ba51-9149ebe4fe2a","Type":"ContainerStarted","Data":"c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8"}
Sep 16 19:14:52.405235 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:52.404829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" event={"ID":"1badcca0-41a4-424d-ba51-9149ebe4fe2a","Type":"ContainerStarted","Data":"5ed38761b3da8cdba4d07e6753f6ec45312a33be84843d37136e5fbb6ae6e955"}
Sep 16 19:14:52.406482 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:52.406457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" event={"ID":"8a53c5e3-950f-4143-aa99-895524dc53bd","Type":"ContainerStarted","Data":"099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c"}
Sep 16 19:14:52.406632 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:52.406496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" event={"ID":"8a53c5e3-950f-4143-aa99-895524dc53bd","Type":"ContainerStarted","Data":"aa3aa5d4c7bc2f6d3a1edfc7224090e183009997df8fa06ee628e62b6d327d94"}
Sep 16 19:14:52.406632 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:52.406603    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:14:52.449282 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:52.449162    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" podStartSLOduration=1.449140943 podStartE2EDuration="1.449140943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:52.448088843 +0000 UTC m=+2134.236417234" watchObservedRunningTime="2026-09-16 19:14:52.449140943 +0000 UTC m=+2134.237469310"
Sep 16 19:14:54.419297 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:54.419261    2561 generic.go:356] "Generic (PLEG): container finished" podID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" containerID="c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8" exitCode=0
Sep 16 19:14:54.419730 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:54.419340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" event={"ID":"1badcca0-41a4-424d-ba51-9149ebe4fe2a","Type":"ContainerDied","Data":"c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8"}
Sep 16 19:14:55.425404 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:55.425364    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" event={"ID":"1badcca0-41a4-424d-ba51-9149ebe4fe2a","Type":"ContainerStarted","Data":"237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3"}
Sep 16 19:14:55.463237 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:55.463174    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" podStartSLOduration=2.245851592 podStartE2EDuration="4.46315669s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.217305098s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:55.461050594 +0000 UTC m=+2137.249378973" watchObservedRunningTime="2026-09-16 19:14:55.46315669 +0000 UTC m=+2137.251485056"
Sep 16 19:14:55.606448 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:14:55.606406    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:15:01.606498 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:01.606451    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:15:01.619154 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:01.619120    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:15:02.466886 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:02.466856    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:15:23.413872 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:23.413792    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:15:37.432493 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.432456    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"]
Sep 16 19:15:37.449072 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.449039    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"]
Sep 16 19:15:37.458376 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:15:37.458324    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:15:37.458376 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:37.458376 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:37.458376 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3" cmd=["/bin/sleep","15"]
Sep 16 19:15:37.458732 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.458389    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" message="" err=<
Sep 16 19:15:37.458732 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:37.458732 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:37.458732 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 19:15:37.458732 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:15:37.458407    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:15:37.458732 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:37.458732 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:37.458732 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" podUID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" containerName="main" containerID="cri-o://237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3"
Sep 16 19:15:37.458732 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.458452    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" podUID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" containerName="main" containerID="cri-o://237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3" gracePeriod=60
Sep 16 19:15:37.807562 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.807534    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:15:37.884318 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884279    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-dshm\") pod \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") "
Sep 16 19:15:37.884521 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884346    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tmp-dir\") pod \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") "
Sep 16 19:15:37.884521 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884396    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1badcca0-41a4-424d-ba51-9149ebe4fe2a-cabundle-cert\") pod \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") "
Sep 16 19:15:37.884521 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884440    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmqp2\" (UniqueName: \"kubernetes.io/projected/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kube-api-access-hmqp2\") pod \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") "
Sep 16 19:15:37.884521 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884467    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tls-certs\") pod \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") "
Sep 16 19:15:37.884521 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-model-cache\") pod \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") "
Sep 16 19:15:37.884826 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884526    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kserve-provision-location\") pod \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") "
Sep 16 19:15:37.884826 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884595    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-home\") pod \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\" (UID: \"1badcca0-41a4-424d-ba51-9149ebe4fe2a\") "
Sep 16 19:15:37.884826 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884647    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1badcca0-41a4-424d-ba51-9149ebe4fe2a" (UID: "1badcca0-41a4-424d-ba51-9149ebe4fe2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:37.884973 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884859    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1badcca0-41a4-424d-ba51-9149ebe4fe2a" (UID: "1badcca0-41a4-424d-ba51-9149ebe4fe2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:37.884973 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884938    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:15:37.885067 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.884969    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-home" (OuterVolumeSpecName: "home") pod "1badcca0-41a4-424d-ba51-9149ebe4fe2a" (UID: "1badcca0-41a4-424d-ba51-9149ebe4fe2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:37.885067 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.885052    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1badcca0-41a4-424d-ba51-9149ebe4fe2a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1badcca0-41a4-424d-ba51-9149ebe4fe2a" (UID: "1badcca0-41a4-424d-ba51-9149ebe4fe2a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:37.886785 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.886757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-dshm" (OuterVolumeSpecName: "dshm") pod "1badcca0-41a4-424d-ba51-9149ebe4fe2a" (UID: "1badcca0-41a4-424d-ba51-9149ebe4fe2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:37.886904 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.886787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1badcca0-41a4-424d-ba51-9149ebe4fe2a" (UID: "1badcca0-41a4-424d-ba51-9149ebe4fe2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:37.886904 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.886867    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kube-api-access-hmqp2" (OuterVolumeSpecName: "kube-api-access-hmqp2") pod "1badcca0-41a4-424d-ba51-9149ebe4fe2a" (UID: "1badcca0-41a4-424d-ba51-9149ebe4fe2a"). InnerVolumeSpecName "kube-api-access-hmqp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:37.904595 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.904547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1badcca0-41a4-424d-ba51-9149ebe4fe2a" (UID: "1badcca0-41a4-424d-ba51-9149ebe4fe2a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:37.985831 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.985791    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:15:37.985831 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.985827    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1badcca0-41a4-424d-ba51-9149ebe4fe2a-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:15:37.986035 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.985838    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmqp2\" (UniqueName: \"kubernetes.io/projected/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kube-api-access-hmqp2\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:15:37.986035 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.985847    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1badcca0-41a4-424d-ba51-9149ebe4fe2a-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:15:37.986035 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.985855    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:15:37.986035 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.985863    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:15:37.986035 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:37.985872    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1badcca0-41a4-424d-ba51-9149ebe4fe2a-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:15:38.626878 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.626843    2561 generic.go:356] "Generic (PLEG): container finished" podID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" containerID="237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3" exitCode=0
Sep 16 19:15:38.627304 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.626892    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"
Sep 16 19:15:38.627304 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.626911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" event={"ID":"1badcca0-41a4-424d-ba51-9149ebe4fe2a","Type":"ContainerDied","Data":"237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3"}
Sep 16 19:15:38.627304 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.626958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv" event={"ID":"1badcca0-41a4-424d-ba51-9149ebe4fe2a","Type":"ContainerDied","Data":"5ed38761b3da8cdba4d07e6753f6ec45312a33be84843d37136e5fbb6ae6e955"}
Sep 16 19:15:38.627304 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.626981    2561 scope.go:117] "RemoveContainer" containerID="237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3"
Sep 16 19:15:38.636786 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.636769    2561 scope.go:117] "RemoveContainer" containerID="c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8"
Sep 16 19:15:38.644505 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.644482    2561 scope.go:117] "RemoveContainer" containerID="237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3"
Sep 16 19:15:38.644839 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:15:38.644820    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3\": container with ID starting with 237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3 not found: ID does not exist" containerID="237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3"
Sep 16 19:15:38.644897 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.644850    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3"} err="failed to get container status \"237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3\": rpc error: code = NotFound desc = could not find container \"237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3\": container with ID starting with 237525a6b5ba53511ba424e03189f4dba39b6bdf27375e0988a8c79fbf8ef7b3 not found: ID does not exist"
Sep 16 19:15:38.644897 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.644868    2561 scope.go:117] "RemoveContainer" containerID="c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8"
Sep 16 19:15:38.645126 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:15:38.645106    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8\": container with ID starting with c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8 not found: ID does not exist" containerID="c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8"
Sep 16 19:15:38.645171 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.645131    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8"} err="failed to get container status \"c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8\": rpc error: code = NotFound desc = could not find container \"c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8\": container with ID starting with c0c8917ac75654fba315e34fe5d893baee6734693285bf4b4b1386daf16a85f8 not found: ID does not exist"
Sep 16 19:15:38.651065 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.651035    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"]
Sep 16 19:15:38.656270 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.656245    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-668bcccbb-hszdv"]
Sep 16 19:15:38.749051 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:38.749016    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" path="/var/lib/kubelet/pods/1badcca0-41a4-424d-ba51-9149ebe4fe2a/volumes"
Sep 16 19:15:43.414193 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:43.414153    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" podUID="8a53c5e3-950f-4143-aa99-895524dc53bd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:15:52.452604 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:52.452527    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" podUID="8a53c5e3-950f-4143-aa99-895524dc53bd" containerName="main" containerID="cri-o://099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c" gracePeriod=45
Sep 16 19:15:53.413474 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:15:53.413434    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" podUID="8a53c5e3-950f-4143-aa99-895524dc53bd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:03.413711 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:03.413671    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" podUID="8a53c5e3-950f-4143-aa99-895524dc53bd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:03.414097 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:03.413793    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:16:13.413997 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:13.413951    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" podUID="8a53c5e3-950f-4143-aa99-895524dc53bd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:22.703444 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.703412    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:16:22.753006 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.752974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a53c5e3-950f-4143-aa99-895524dc53bd-tls-certs\") pod \"8a53c5e3-950f-4143-aa99-895524dc53bd\" (UID: \"8a53c5e3-950f-4143-aa99-895524dc53bd\") "
Sep 16 19:16:22.753171 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.753056    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88l5t\" (UniqueName: \"kubernetes.io/projected/8a53c5e3-950f-4143-aa99-895524dc53bd-kube-api-access-88l5t\") pod \"8a53c5e3-950f-4143-aa99-895524dc53bd\" (UID: \"8a53c5e3-950f-4143-aa99-895524dc53bd\") "
Sep 16 19:16:22.755140 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.755110    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a53c5e3-950f-4143-aa99-895524dc53bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a53c5e3-950f-4143-aa99-895524dc53bd" (UID: "8a53c5e3-950f-4143-aa99-895524dc53bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:22.755259 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.755237    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a53c5e3-950f-4143-aa99-895524dc53bd-kube-api-access-88l5t" (OuterVolumeSpecName: "kube-api-access-88l5t") pod "8a53c5e3-950f-4143-aa99-895524dc53bd" (UID: "8a53c5e3-950f-4143-aa99-895524dc53bd"). InnerVolumeSpecName "kube-api-access-88l5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:22.823365 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.823324    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a53c5e3-950f-4143-aa99-895524dc53bd" containerID="099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c" exitCode=0
Sep 16 19:16:22.823595 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.823374    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"
Sep 16 19:16:22.823595 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.823379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" event={"ID":"8a53c5e3-950f-4143-aa99-895524dc53bd","Type":"ContainerDied","Data":"099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c"}
Sep 16 19:16:22.823595 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.823452    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj" event={"ID":"8a53c5e3-950f-4143-aa99-895524dc53bd","Type":"ContainerDied","Data":"aa3aa5d4c7bc2f6d3a1edfc7224090e183009997df8fa06ee628e62b6d327d94"}
Sep 16 19:16:22.823595 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.823488    2561 scope.go:117] "RemoveContainer" containerID="099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c"
Sep 16 19:16:22.832503 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.832483    2561 scope.go:117] "RemoveContainer" containerID="099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c"
Sep 16 19:16:22.832815 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:16:22.832792    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c\": container with ID starting with 099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c not found: ID does not exist" containerID="099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c"
Sep 16 19:16:22.832894 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.832824    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c"} err="failed to get container status \"099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c\": rpc error: code = NotFound desc = could not find container \"099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c\": container with ID starting with 099785b0418f8567c9f49710cabfeb78bdad2af281bbbce030c7778c3c14c06c not found: ID does not exist"
Sep 16 19:16:22.845410 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.845374    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"]
Sep 16 19:16:22.849212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.849182    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785m48lj"]
Sep 16 19:16:22.853722 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.853695    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88l5t\" (UniqueName: \"kubernetes.io/projected/8a53c5e3-950f-4143-aa99-895524dc53bd-kube-api-access-88l5t\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.853722 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:22.853724    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a53c5e3-950f-4143-aa99-895524dc53bd-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.755391 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:24.755355    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a53c5e3-950f-4143-aa99-895524dc53bd" path="/var/lib/kubelet/pods/8a53c5e3-950f-4143-aa99-895524dc53bd/volumes"
Sep 16 19:16:37.794885 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.794842    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"]
Sep 16 19:16:37.795299 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.795165    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" containerName="storage-initializer"
Sep 16 19:16:37.795299 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.795176    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" containerName="storage-initializer"
Sep 16 19:16:37.795299 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.795190    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a53c5e3-950f-4143-aa99-895524dc53bd" containerName="main"
Sep 16 19:16:37.795299 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.795197    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a53c5e3-950f-4143-aa99-895524dc53bd" containerName="main"
Sep 16 19:16:37.795299 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.795220    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" containerName="main"
Sep 16 19:16:37.795299 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.795226    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" containerName="main"
Sep 16 19:16:37.795299 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.795282    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a53c5e3-950f-4143-aa99-895524dc53bd" containerName="main"
Sep 16 19:16:37.795299 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.795293    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1badcca0-41a4-424d-ba51-9149ebe4fe2a" containerName="main"
Sep 16 19:16:37.799865 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.799835    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.802287 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.802263    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:16:37.802428 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.802396    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:16:37.802428 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.802396    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:16:37.802554 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.802454    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:16:37.803112 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.803095    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:16:37.803159 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.803136    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-kdv7p\""
Sep 16 19:16:37.812418 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.812390    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"]
Sep 16 19:16:37.882704 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.882666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkt4n\" (UniqueName: \"kubernetes.io/projected/de872c14-2108-425f-aa8b-aaa588a60d6e-kube-api-access-wkt4n\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.882704 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.882708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-home\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.882997 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.882782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-model-cache\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.882997 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.882829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de872c14-2108-425f-aa8b-aaa588a60d6e-tls-certs\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.882997 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.882893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de872c14-2108-425f-aa8b-aaa588a60d6e-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.882997 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.882943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-dshm\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.882997 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.882977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.883180 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.883005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-tmp-dir\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984028 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.983979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de872c14-2108-425f-aa8b-aaa588a60d6e-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984028 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-dshm\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984293 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984293 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-tmp-dir\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984293 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkt4n\" (UniqueName: \"kubernetes.io/projected/de872c14-2108-425f-aa8b-aaa588a60d6e-kube-api-access-wkt4n\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984456 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-home\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984456 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984356    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-model-cache\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984456 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de872c14-2108-425f-aa8b-aaa588a60d6e-tls-certs\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984658 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984531    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-tmp-dir\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984658 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-home\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984844 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-model-cache\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984914 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de872c14-2108-425f-aa8b-aaa588a60d6e-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.984914 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.984881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.986472 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.986450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-dshm\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.986745 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.986729    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de872c14-2108-425f-aa8b-aaa588a60d6e-tls-certs\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:37.994466 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:37.994446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkt4n\" (UniqueName: \"kubernetes.io/projected/de872c14-2108-425f-aa8b-aaa588a60d6e-kube-api-access-wkt4n\") pod \"tokenizer-template-args-test-kserve-67c45f5458-j6x8n\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:38.108031 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.107956    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"]
Sep 16 19:16:38.110682 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.110657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:38.112373 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.112356    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:16:38.114859 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.114837    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-b4rbt\""
Sep 16 19:16:38.127261 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.127201    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"]
Sep 16 19:16:38.185702 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.185669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm6v6\" (UniqueName: \"kubernetes.io/projected/5d095039-47f2-4c97-9f4c-19886fa22b2a-kube-api-access-rm6v6\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh\" (UID: \"5d095039-47f2-4c97-9f4c-19886fa22b2a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:16:38.185857 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.185733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d095039-47f2-4c97-9f4c-19886fa22b2a-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh\" (UID: \"5d095039-47f2-4c97-9f4c-19886fa22b2a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:16:38.287008 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.286970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm6v6\" (UniqueName: \"kubernetes.io/projected/5d095039-47f2-4c97-9f4c-19886fa22b2a-kube-api-access-rm6v6\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh\" (UID: \"5d095039-47f2-4c97-9f4c-19886fa22b2a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:16:38.287008 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.287025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d095039-47f2-4c97-9f4c-19886fa22b2a-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh\" (UID: \"5d095039-47f2-4c97-9f4c-19886fa22b2a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:16:38.289493 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.289472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d095039-47f2-4c97-9f4c-19886fa22b2a-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh\" (UID: \"5d095039-47f2-4c97-9f4c-19886fa22b2a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:16:38.295390 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.295367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm6v6\" (UniqueName: \"kubernetes.io/projected/5d095039-47f2-4c97-9f4c-19886fa22b2a-kube-api-access-rm6v6\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh\" (UID: \"5d095039-47f2-4c97-9f4c-19886fa22b2a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:16:38.390776 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.390554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"]
Sep 16 19:16:38.393474 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:16:38.393443    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde872c14_2108_425f_aa8b_aaa588a60d6e.slice/crio-7d84e695edc08b2f087b6f0f763c050b5e5407dda340193e85e798f87187e67f WatchSource:0}: Error finding container 7d84e695edc08b2f087b6f0f763c050b5e5407dda340193e85e798f87187e67f: Status 404 returned error can't find the container with id 7d84e695edc08b2f087b6f0f763c050b5e5407dda340193e85e798f87187e67f
Sep 16 19:16:38.438881 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.438852    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:16:38.716318 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.716292    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"]
Sep 16 19:16:38.718352 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:16:38.718325    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d095039_47f2_4c97_9f4c_19886fa22b2a.slice/crio-e4b3d7410537d2761f4053e07a762d337c6150f7eefc676d7ab291caea5a616c WatchSource:0}: Error finding container e4b3d7410537d2761f4053e07a762d337c6150f7eefc676d7ab291caea5a616c: Status 404 returned error can't find the container with id e4b3d7410537d2761f4053e07a762d337c6150f7eefc676d7ab291caea5a616c
Sep 16 19:16:38.903911 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.903875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" event={"ID":"5d095039-47f2-4c97-9f4c-19886fa22b2a","Type":"ContainerStarted","Data":"6c977cbf871f16b8fc25af2312f8e48d44466de00159828ada8ceae3f21851c6"}
Sep 16 19:16:38.904355 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.903931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" event={"ID":"5d095039-47f2-4c97-9f4c-19886fa22b2a","Type":"ContainerStarted","Data":"e4b3d7410537d2761f4053e07a762d337c6150f7eefc676d7ab291caea5a616c"}
Sep 16 19:16:38.904355 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.903985    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:16:38.905625 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.905597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" event={"ID":"de872c14-2108-425f-aa8b-aaa588a60d6e","Type":"ContainerStarted","Data":"32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8"}
Sep 16 19:16:38.905751 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.905652    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" event={"ID":"de872c14-2108-425f-aa8b-aaa588a60d6e","Type":"ContainerStarted","Data":"7d84e695edc08b2f087b6f0f763c050b5e5407dda340193e85e798f87187e67f"}
Sep 16 19:16:38.958113 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:38.958060    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" podStartSLOduration=0.958044558 podStartE2EDuration="958.044558ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:38.926633949 +0000 UTC m=+2240.714962342" watchObservedRunningTime="2026-09-16 19:16:38.958044558 +0000 UTC m=+2240.746372923"
Sep 16 19:16:40.916015 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:40.915933    2561 generic.go:356] "Generic (PLEG): container finished" podID="de872c14-2108-425f-aa8b-aaa588a60d6e" containerID="32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8" exitCode=0
Sep 16 19:16:40.916403 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:40.916011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" event={"ID":"de872c14-2108-425f-aa8b-aaa588a60d6e","Type":"ContainerDied","Data":"32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8"}
Sep 16 19:16:41.923081 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:41.923045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" event={"ID":"de872c14-2108-425f-aa8b-aaa588a60d6e","Type":"ContainerStarted","Data":"a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee"}
Sep 16 19:16:41.945560 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:41.945499    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" podStartSLOduration=2.689049382 podStartE2EDuration="4.945479356s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.256429974s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:41.945240981 +0000 UTC m=+2243.733569347" watchObservedRunningTime="2026-09-16 19:16:41.945479356 +0000 UTC m=+2243.733807723"
Sep 16 19:16:42.110944 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:42.110911    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:48.111619 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:48.111583    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:48.124266 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:48.124238    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:16:48.973061 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:16:48.973027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:17:09.911895 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:09.911859    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:17:51.255839 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.255796    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"]
Sep 16 19:17:51.257314 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.257288    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"]
Sep 16 19:17:51.264838 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:17:51.264806    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:17:51.264838 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:51.264838 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:51.264838 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee" cmd=["/bin/sleep","15"]
Sep 16 19:17:51.265063 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.264853    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" message="" err=<
Sep 16 19:17:51.265063 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:51.265063 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:51.265063 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 19:17:51.265063 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:17:51.264865    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:17:51.265063 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:51.265063 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:51.265063 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" podUID="de872c14-2108-425f-aa8b-aaa588a60d6e" containerName="main" containerID="cri-o://a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee"
Sep 16 19:17:51.265063 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.264898    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" podUID="de872c14-2108-425f-aa8b-aaa588a60d6e" containerName="main" containerID="cri-o://a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee" gracePeriod=60
Sep 16 19:17:51.619820 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.619792    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:17:51.648209 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648176    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-dshm\") pod \"de872c14-2108-425f-aa8b-aaa588a60d6e\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") "
Sep 16 19:17:51.648405 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648241    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-kserve-provision-location\") pod \"de872c14-2108-425f-aa8b-aaa588a60d6e\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") "
Sep 16 19:17:51.648405 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648288    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de872c14-2108-425f-aa8b-aaa588a60d6e-cabundle-cert\") pod \"de872c14-2108-425f-aa8b-aaa588a60d6e\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") "
Sep 16 19:17:51.648405 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648314    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-home\") pod \"de872c14-2108-425f-aa8b-aaa588a60d6e\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") "
Sep 16 19:17:51.648405 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648372    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-tmp-dir\") pod \"de872c14-2108-425f-aa8b-aaa588a60d6e\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") "
Sep 16 19:17:51.648657 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648416    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de872c14-2108-425f-aa8b-aaa588a60d6e-tls-certs\") pod \"de872c14-2108-425f-aa8b-aaa588a60d6e\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") "
Sep 16 19:17:51.648657 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648453    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkt4n\" (UniqueName: \"kubernetes.io/projected/de872c14-2108-425f-aa8b-aaa588a60d6e-kube-api-access-wkt4n\") pod \"de872c14-2108-425f-aa8b-aaa588a60d6e\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") "
Sep 16 19:17:51.648657 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-model-cache\") pod \"de872c14-2108-425f-aa8b-aaa588a60d6e\" (UID: \"de872c14-2108-425f-aa8b-aaa588a60d6e\") "
Sep 16 19:17:51.648657 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-home" (OuterVolumeSpecName: "home") pod "de872c14-2108-425f-aa8b-aaa588a60d6e" (UID: "de872c14-2108-425f-aa8b-aaa588a60d6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.648878 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648656    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de872c14-2108-425f-aa8b-aaa588a60d6e" (UID: "de872c14-2108-425f-aa8b-aaa588a60d6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.648878 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648775    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de872c14-2108-425f-aa8b-aaa588a60d6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de872c14-2108-425f-aa8b-aaa588a60d6e" (UID: "de872c14-2108-425f-aa8b-aaa588a60d6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:51.648878 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648792    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "de872c14-2108-425f-aa8b-aaa588a60d6e" (UID: "de872c14-2108-425f-aa8b-aaa588a60d6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.648878 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648830    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.648878 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.648851    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.650910 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.650879    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de872c14-2108-425f-aa8b-aaa588a60d6e-kube-api-access-wkt4n" (OuterVolumeSpecName: "kube-api-access-wkt4n") pod "de872c14-2108-425f-aa8b-aaa588a60d6e" (UID: "de872c14-2108-425f-aa8b-aaa588a60d6e"). InnerVolumeSpecName "kube-api-access-wkt4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:51.651052 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.650906    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-dshm" (OuterVolumeSpecName: "dshm") pod "de872c14-2108-425f-aa8b-aaa588a60d6e" (UID: "de872c14-2108-425f-aa8b-aaa588a60d6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.651052 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.650999    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de872c14-2108-425f-aa8b-aaa588a60d6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de872c14-2108-425f-aa8b-aaa588a60d6e" (UID: "de872c14-2108-425f-aa8b-aaa588a60d6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:51.669000 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.668952    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de872c14-2108-425f-aa8b-aaa588a60d6e" (UID: "de872c14-2108-425f-aa8b-aaa588a60d6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:51.749654 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.749608    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de872c14-2108-425f-aa8b-aaa588a60d6e-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.749654 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.749647    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkt4n\" (UniqueName: \"kubernetes.io/projected/de872c14-2108-425f-aa8b-aaa588a60d6e-kube-api-access-wkt4n\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.749654 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.749657    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.749654 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.749665    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.749932 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.749674    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de872c14-2108-425f-aa8b-aaa588a60d6e-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.749932 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:51.749682    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de872c14-2108-425f-aa8b-aaa588a60d6e-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:17:52.262030 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.261992    2561 generic.go:356] "Generic (PLEG): container finished" podID="de872c14-2108-425f-aa8b-aaa588a60d6e" containerID="a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee" exitCode=0
Sep 16 19:17:52.262455 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.262044    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"
Sep 16 19:17:52.262455 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.262059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" event={"ID":"de872c14-2108-425f-aa8b-aaa588a60d6e","Type":"ContainerDied","Data":"a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee"}
Sep 16 19:17:52.262455 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.262103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n" event={"ID":"de872c14-2108-425f-aa8b-aaa588a60d6e","Type":"ContainerDied","Data":"7d84e695edc08b2f087b6f0f763c050b5e5407dda340193e85e798f87187e67f"}
Sep 16 19:17:52.262455 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.262132    2561 scope.go:117] "RemoveContainer" containerID="a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee"
Sep 16 19:17:52.271150 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.270982    2561 scope.go:117] "RemoveContainer" containerID="32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8"
Sep 16 19:17:52.279724 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.279695    2561 scope.go:117] "RemoveContainer" containerID="a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee"
Sep 16 19:17:52.280070 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:17:52.280044    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee\": container with ID starting with a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee not found: ID does not exist" containerID="a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee"
Sep 16 19:17:52.280222 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.280080    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee"} err="failed to get container status \"a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee\": rpc error: code = NotFound desc = could not find container \"a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee\": container with ID starting with a97390547bab064bc05418ddedc79ccef25c9be6ec234b049dc92cc0d408b4ee not found: ID does not exist"
Sep 16 19:17:52.280222 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.280103    2561 scope.go:117] "RemoveContainer" containerID="32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8"
Sep 16 19:17:52.280454 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:17:52.280435    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8\": container with ID starting with 32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8 not found: ID does not exist" containerID="32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8"
Sep 16 19:17:52.280495 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.280462    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8"} err="failed to get container status \"32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8\": rpc error: code = NotFound desc = could not find container \"32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8\": container with ID starting with 32684c9e5abf6ddcfc33305843269d58a9816033d25cf1fe1bb76bf0e7daa4f8 not found: ID does not exist"
Sep 16 19:17:52.290319 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.290294    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"]
Sep 16 19:17:52.302622 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.302592    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-67c45f5458-j6x8n"]
Sep 16 19:17:52.749405 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:52.749359    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de872c14-2108-425f-aa8b-aaa588a60d6e" path="/var/lib/kubelet/pods/de872c14-2108-425f-aa8b-aaa588a60d6e/volumes"
Sep 16 19:17:59.912293 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:17:59.912247    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" podUID="5d095039-47f2-4c97-9f4c-19886fa22b2a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:06.276354 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:06.276296    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" podUID="5d095039-47f2-4c97-9f4c-19886fa22b2a" containerName="main" containerID="cri-o://6c977cbf871f16b8fc25af2312f8e48d44466de00159828ada8ceae3f21851c6" gracePeriod=45
Sep 16 19:18:09.912550 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:09.912505    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" podUID="5d095039-47f2-4c97-9f4c-19886fa22b2a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:19.912840 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:19.912752    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" podUID="5d095039-47f2-4c97-9f4c-19886fa22b2a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:19.913199 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:19.912867    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:18:29.912396 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:29.912352    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" podUID="5d095039-47f2-4c97-9f4c-19886fa22b2a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:36.475100 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:36.475066    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d095039-47f2-4c97-9f4c-19886fa22b2a" containerID="6c977cbf871f16b8fc25af2312f8e48d44466de00159828ada8ceae3f21851c6" exitCode=0
Sep 16 19:18:36.475449 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:36.475141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" event={"ID":"5d095039-47f2-4c97-9f4c-19886fa22b2a","Type":"ContainerDied","Data":"6c977cbf871f16b8fc25af2312f8e48d44466de00159828ada8ceae3f21851c6"}
Sep 16 19:18:36.525410 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:36.525333    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:18:36.540796 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:36.540732    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm6v6\" (UniqueName: \"kubernetes.io/projected/5d095039-47f2-4c97-9f4c-19886fa22b2a-kube-api-access-rm6v6\") pod \"5d095039-47f2-4c97-9f4c-19886fa22b2a\" (UID: \"5d095039-47f2-4c97-9f4c-19886fa22b2a\") "
Sep 16 19:18:36.540939 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:36.540816    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d095039-47f2-4c97-9f4c-19886fa22b2a-tls-certs\") pod \"5d095039-47f2-4c97-9f4c-19886fa22b2a\" (UID: \"5d095039-47f2-4c97-9f4c-19886fa22b2a\") "
Sep 16 19:18:36.542904 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:36.542867    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d095039-47f2-4c97-9f4c-19886fa22b2a-kube-api-access-rm6v6" (OuterVolumeSpecName: "kube-api-access-rm6v6") pod "5d095039-47f2-4c97-9f4c-19886fa22b2a" (UID: "5d095039-47f2-4c97-9f4c-19886fa22b2a"). InnerVolumeSpecName "kube-api-access-rm6v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:36.543013 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:36.542910    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d095039-47f2-4c97-9f4c-19886fa22b2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d095039-47f2-4c97-9f4c-19886fa22b2a" (UID: "5d095039-47f2-4c97-9f4c-19886fa22b2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:36.642008 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:36.641966    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d095039-47f2-4c97-9f4c-19886fa22b2a-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:18:36.642008 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:36.641998    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm6v6\" (UniqueName: \"kubernetes.io/projected/5d095039-47f2-4c97-9f4c-19886fa22b2a-kube-api-access-rm6v6\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:18:37.481343 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:37.481303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh" event={"ID":"5d095039-47f2-4c97-9f4c-19886fa22b2a","Type":"ContainerDied","Data":"e4b3d7410537d2761f4053e07a762d337c6150f7eefc676d7ab291caea5a616c"}
Sep 16 19:18:37.481343 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:37.481339    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"
Sep 16 19:18:37.481865 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:37.481391    2561 scope.go:117] "RemoveContainer" containerID="6c977cbf871f16b8fc25af2312f8e48d44466de00159828ada8ceae3f21851c6"
Sep 16 19:18:37.500265 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:37.500237    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"]
Sep 16 19:18:37.503991 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:37.503967    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d4rzlh"]
Sep 16 19:18:38.749099 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:38.749053    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d095039-47f2-4c97-9f4c-19886fa22b2a" path="/var/lib/kubelet/pods/5d095039-47f2-4c97-9f4c-19886fa22b2a/volumes"
Sep 16 19:18:55.776584 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.776544    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"]
Sep 16 19:18:55.776977 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.776908    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d095039-47f2-4c97-9f4c-19886fa22b2a" containerName="main"
Sep 16 19:18:55.776977 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.776921    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d095039-47f2-4c97-9f4c-19886fa22b2a" containerName="main"
Sep 16 19:18:55.776977 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.776932    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de872c14-2108-425f-aa8b-aaa588a60d6e" containerName="main"
Sep 16 19:18:55.776977 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.776938    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="de872c14-2108-425f-aa8b-aaa588a60d6e" containerName="main"
Sep 16 19:18:55.776977 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.776950    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de872c14-2108-425f-aa8b-aaa588a60d6e" containerName="storage-initializer"
Sep 16 19:18:55.776977 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.776955    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="de872c14-2108-425f-aa8b-aaa588a60d6e" containerName="storage-initializer"
Sep 16 19:18:55.777180 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.777013    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d095039-47f2-4c97-9f4c-19886fa22b2a" containerName="main"
Sep 16 19:18:55.777180 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.777026    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="de872c14-2108-425f-aa8b-aaa588a60d6e" containerName="main"
Sep 16 19:18:55.779954 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.779933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:18:55.783166 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.783142    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-9mtxm\""
Sep 16 19:18:55.783166 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.783154    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:18:55.783353 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.783196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:18:55.783353 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.783276    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:18:55.783353 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.783328    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-95897\""
Sep 16 19:18:55.788965 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.788940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"]
Sep 16 19:18:55.905328 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.905281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzpt\" (UniqueName: \"kubernetes.io/projected/ea264a97-00e6-4c90-a217-d4bb25a62d0c-kube-api-access-6dzpt\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl\" (UID: \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:18:55.905518 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:55.905339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea264a97-00e6-4c90-a217-d4bb25a62d0c-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl\" (UID: \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:18:56.006834 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.006793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzpt\" (UniqueName: \"kubernetes.io/projected/ea264a97-00e6-4c90-a217-d4bb25a62d0c-kube-api-access-6dzpt\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl\" (UID: \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:18:56.007089 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.006855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea264a97-00e6-4c90-a217-d4bb25a62d0c-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl\" (UID: \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:18:56.009339 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.009309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea264a97-00e6-4c90-a217-d4bb25a62d0c-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl\" (UID: \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:18:56.015992 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.015966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzpt\" (UniqueName: \"kubernetes.io/projected/ea264a97-00e6-4c90-a217-d4bb25a62d0c-kube-api-access-6dzpt\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl\" (UID: \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:18:56.090540 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.090460    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:18:56.373041 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.373015    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"]
Sep 16 19:18:56.375331 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:18:56.375301    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea264a97_00e6_4c90_a217_d4bb25a62d0c.slice/crio-cb3f71c001179cbcc10c12dc9b153bc0df5af054baf49e54a723744d92fc1843 WatchSource:0}: Error finding container cb3f71c001179cbcc10c12dc9b153bc0df5af054baf49e54a723744d92fc1843: Status 404 returned error can't find the container with id cb3f71c001179cbcc10c12dc9b153bc0df5af054baf49e54a723744d92fc1843
Sep 16 19:18:56.574328 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.574289    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" event={"ID":"ea264a97-00e6-4c90-a217-d4bb25a62d0c","Type":"ContainerStarted","Data":"f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88"}
Sep 16 19:18:56.574544 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.574366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" event={"ID":"ea264a97-00e6-4c90-a217-d4bb25a62d0c","Type":"ContainerStarted","Data":"cb3f71c001179cbcc10c12dc9b153bc0df5af054baf49e54a723744d92fc1843"}
Sep 16 19:18:56.574544 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.574442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:18:56.598177 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:18:56.598120    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" podStartSLOduration=1.5981037200000001 podStartE2EDuration="1.59810372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:56.595623583 +0000 UTC m=+2378.383951949" watchObservedRunningTime="2026-09-16 19:18:56.59810372 +0000 UTC m=+2378.386432085"
Sep 16 19:19:18.850166 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:19:18.850136    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:19:18.853892 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:19:18.853868    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:19:27.580809 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:19:27.580779    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:19:29.363429 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:19:29.363385    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"]
Sep 16 19:19:37.580401 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:19:37.580360    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" podUID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:44.382376 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:19:44.382262    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" podUID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" containerName="main" containerID="cri-o://f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88" gracePeriod=45
Sep 16 19:19:47.580858 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:19:47.580814    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" podUID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:57.580495 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:19:57.580454    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" podUID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:57.580995 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:19:57.580561    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:20:07.580352 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:07.580313    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" podUID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:20:14.626868 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.626841    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:20:14.697592 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.697475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dzpt\" (UniqueName: \"kubernetes.io/projected/ea264a97-00e6-4c90-a217-d4bb25a62d0c-kube-api-access-6dzpt\") pod \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\" (UID: \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\") "
Sep 16 19:20:14.697592 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.697515    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea264a97-00e6-4c90-a217-d4bb25a62d0c-tls-certs\") pod \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\" (UID: \"ea264a97-00e6-4c90-a217-d4bb25a62d0c\") "
Sep 16 19:20:14.699711 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.699689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea264a97-00e6-4c90-a217-d4bb25a62d0c-kube-api-access-6dzpt" (OuterVolumeSpecName: "kube-api-access-6dzpt") pod "ea264a97-00e6-4c90-a217-d4bb25a62d0c" (UID: "ea264a97-00e6-4c90-a217-d4bb25a62d0c"). InnerVolumeSpecName "kube-api-access-6dzpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:14.699800 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.699778    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea264a97-00e6-4c90-a217-d4bb25a62d0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea264a97-00e6-4c90-a217-d4bb25a62d0c" (UID: "ea264a97-00e6-4c90-a217-d4bb25a62d0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:14.798381 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.798346    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzpt\" (UniqueName: \"kubernetes.io/projected/ea264a97-00e6-4c90-a217-d4bb25a62d0c-kube-api-access-6dzpt\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:20:14.798381 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.798374    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea264a97-00e6-4c90-a217-d4bb25a62d0c-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:20:14.916585 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.916543    2561 generic.go:356] "Generic (PLEG): container finished" podID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" containerID="f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88" exitCode=0
Sep 16 19:20:14.916759 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.916613    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"
Sep 16 19:20:14.916759 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.916603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" event={"ID":"ea264a97-00e6-4c90-a217-d4bb25a62d0c","Type":"ContainerDied","Data":"f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88"}
Sep 16 19:20:14.916759 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.916738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl" event={"ID":"ea264a97-00e6-4c90-a217-d4bb25a62d0c","Type":"ContainerDied","Data":"cb3f71c001179cbcc10c12dc9b153bc0df5af054baf49e54a723744d92fc1843"}
Sep 16 19:20:14.916931 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.916768    2561 scope.go:117] "RemoveContainer" containerID="f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88"
Sep 16 19:20:14.926262 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.926237    2561 scope.go:117] "RemoveContainer" containerID="f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88"
Sep 16 19:20:14.926538 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:20:14.926521    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88\": container with ID starting with f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88 not found: ID does not exist" containerID="f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88"
Sep 16 19:20:14.926630 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.926553    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88"} err="failed to get container status \"f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88\": rpc error: code = NotFound desc = could not find container \"f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88\": container with ID starting with f8e66bed2b0e142aac56f4b4911013e029150aaf46d8f4706c62320bee353b88 not found: ID does not exist"
Sep 16 19:20:14.937703 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.937679    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"]
Sep 16 19:20:14.943963 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:14.943938    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche9tfcl"]
Sep 16 19:20:16.749515 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:16.749466    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" path="/var/lib/kubelet/pods/ea264a97-00e6-4c90-a217-d4bb25a62d0c/volumes"
Sep 16 19:20:52.282633 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.282601    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"]
Sep 16 19:20:52.283228 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.283104    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" containerName="main"
Sep 16 19:20:52.283228 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.283123    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" containerName="main"
Sep 16 19:20:52.283228 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.283195    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea264a97-00e6-4c90-a217-d4bb25a62d0c" containerName="main"
Sep 16 19:20:52.286238 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.286218    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:20:52.290143 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.290118    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:20:52.290308 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.290283    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-lzzfl\""
Sep 16 19:20:52.290379 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.290341    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:20:52.290379 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.290358    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:20:52.290505 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.290358    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-vhfcs\""
Sep 16 19:20:52.310275 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.310242    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"]
Sep 16 19:20:52.419399 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.419355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmpl\" (UniqueName: \"kubernetes.io/projected/433ed2ac-cbec-45f4-bef6-9652a87cc291-kube-api-access-6zmpl\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d\" (UID: \"433ed2ac-cbec-45f4-bef6-9652a87cc291\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:20:52.419614 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.419465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433ed2ac-cbec-45f4-bef6-9652a87cc291-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d\" (UID: \"433ed2ac-cbec-45f4-bef6-9652a87cc291\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:20:52.520737 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.520701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433ed2ac-cbec-45f4-bef6-9652a87cc291-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d\" (UID: \"433ed2ac-cbec-45f4-bef6-9652a87cc291\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:20:52.522943 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.520770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmpl\" (UniqueName: \"kubernetes.io/projected/433ed2ac-cbec-45f4-bef6-9652a87cc291-kube-api-access-6zmpl\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d\" (UID: \"433ed2ac-cbec-45f4-bef6-9652a87cc291\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:20:52.523349 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.523329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433ed2ac-cbec-45f4-bef6-9652a87cc291-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d\" (UID: \"433ed2ac-cbec-45f4-bef6-9652a87cc291\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:20:52.530261 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.530237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmpl\" (UniqueName: \"kubernetes.io/projected/433ed2ac-cbec-45f4-bef6-9652a87cc291-kube-api-access-6zmpl\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d\" (UID: \"433ed2ac-cbec-45f4-bef6-9652a87cc291\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:20:52.596635 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.596522    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:20:52.876547 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:52.876516    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"]
Sep 16 19:20:52.879415 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:20:52.879386    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ed2ac_cbec_45f4_bef6_9652a87cc291.slice/crio-6893705aae58fde953ddb0590f41a79f0217c16b382152bcbddafcb97fa0fc21 WatchSource:0}: Error finding container 6893705aae58fde953ddb0590f41a79f0217c16b382152bcbddafcb97fa0fc21: Status 404 returned error can't find the container with id 6893705aae58fde953ddb0590f41a79f0217c16b382152bcbddafcb97fa0fc21
Sep 16 19:20:53.086317 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:53.086268    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" event={"ID":"433ed2ac-cbec-45f4-bef6-9652a87cc291","Type":"ContainerStarted","Data":"9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae"}
Sep 16 19:20:53.086526 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:53.086342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" event={"ID":"433ed2ac-cbec-45f4-bef6-9652a87cc291","Type":"ContainerStarted","Data":"6893705aae58fde953ddb0590f41a79f0217c16b382152bcbddafcb97fa0fc21"}
Sep 16 19:20:53.086526 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:53.086414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:20:53.131901 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:20:53.131794    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" podStartSLOduration=1.131777685 podStartE2EDuration="1.131777685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:53.129508688 +0000 UTC m=+2494.917837082" watchObservedRunningTime="2026-09-16 19:20:53.131777685 +0000 UTC m=+2494.920106050"
Sep 16 19:21:24.092948 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:21:24.092869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:22:05.276247 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.276208    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"]
Sep 16 19:22:05.279669 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.279649    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.286677 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.286656    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 19:22:05.286815 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.286687    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 16 19:22:05.286815 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.286720    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 19:22:05.286815 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.286724    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-29p8r\""
Sep 16 19:22:05.303745 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.303708    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"]
Sep 16 19:22:05.341913 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.341872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.341913 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.341914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.342119 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.341940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.342119 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.341962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.342119 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.342011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.342119 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.342029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrr7\" (UniqueName: \"kubernetes.io/projected/f94fe311-68f5-4e17-a22a-02b1cfa015d5-kube-api-access-vnrr7\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.442735 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.442691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.442735 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.442738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrr7\" (UniqueName: \"kubernetes.io/projected/f94fe311-68f5-4e17-a22a-02b1cfa015d5-kube-api-access-vnrr7\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.443007 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.442819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.443007 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.442978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.443131 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.443015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.443131 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.443036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.443237 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.443138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.443364 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.443342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.443419 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.443359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.445239 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.445215    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.445380 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.445362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.453118 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.453095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrr7\" (UniqueName: \"kubernetes.io/projected/f94fe311-68f5-4e17-a22a-02b1cfa015d5-kube-api-access-vnrr7\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.589693 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.589592    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:05.873679 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:05.873606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"]
Sep 16 19:22:05.876386 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:22:05.876358    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94fe311_68f5_4e17_a22a_02b1cfa015d5.slice/crio-21b0579f5968ad9446861b9eb0149169b9adf341aa524162922840dfcf03a32b WatchSource:0}: Error finding container 21b0579f5968ad9446861b9eb0149169b9adf341aa524162922840dfcf03a32b: Status 404 returned error can't find the container with id 21b0579f5968ad9446861b9eb0149169b9adf341aa524162922840dfcf03a32b
Sep 16 19:22:06.408660 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:06.408613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt" event={"ID":"f94fe311-68f5-4e17-a22a-02b1cfa015d5","Type":"ContainerStarted","Data":"93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623"}
Sep 16 19:22:06.409062 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:06.408712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt" event={"ID":"f94fe311-68f5-4e17-a22a-02b1cfa015d5","Type":"ContainerStarted","Data":"21b0579f5968ad9446861b9eb0149169b9adf341aa524162922840dfcf03a32b"}
Sep 16 19:22:06.436431 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:06.436374    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt" podStartSLOduration=1.436358649 podStartE2EDuration="1.436358649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:06.434827842 +0000 UTC m=+2568.223156211" watchObservedRunningTime="2026-09-16 19:22:06.436358649 +0000 UTC m=+2568.224687059"
Sep 16 19:22:06.589985 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:06.589945    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:15.590267 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:15.590224    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:15.602954 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:15.602925    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:16.465398 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:16.465364    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:29.449192 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.449158    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"]
Sep 16 19:22:29.457935 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:22:29.457898    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:22:29.457935 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:22:29.457935 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:22:29.457935 ip-10-0-142-197 kubenswrapper[2561]:  > containerID="93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623" cmd=["/bin/sleep","15"]
Sep 16 19:22:29.458179 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.457950    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt" message="" err=<
Sep 16 19:22:29.458179 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:22:29.458179 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:22:29.458179 ip-10-0-142-197 kubenswrapper[2561]:  >
Sep 16 19:22:29.458179 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:22:29.457967    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:22:29.458179 ip-10-0-142-197 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:22:29.458179 ip-10-0-142-197 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 16 19:22:29.458179 ip-10-0-142-197 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt" podUID="f94fe311-68f5-4e17-a22a-02b1cfa015d5" containerName="main" containerID="cri-o://93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623"
Sep 16 19:22:29.458179 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.458012    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt" podUID="f94fe311-68f5-4e17-a22a-02b1cfa015d5" containerName="main" containerID="cri-o://93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623" gracePeriod=60
Sep 16 19:22:29.811665 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.811634    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:29.855080 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-dshm\") pod \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") "
Sep 16 19:22:29.855255 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855087    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tmp-dir\") pod \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") "
Sep 16 19:22:29.855255 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855119    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnrr7\" (UniqueName: \"kubernetes.io/projected/f94fe311-68f5-4e17-a22a-02b1cfa015d5-kube-api-access-vnrr7\") pod \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") "
Sep 16 19:22:29.855255 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-model-cache\") pod \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") "
Sep 16 19:22:29.855255 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855211    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-home\") pod \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") "
Sep 16 19:22:29.855255 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855242    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tls-certs\") pod \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\" (UID: \"f94fe311-68f5-4e17-a22a-02b1cfa015d5\") "
Sep 16 19:22:29.855534 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855385    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f94fe311-68f5-4e17-a22a-02b1cfa015d5" (UID: "f94fe311-68f5-4e17-a22a-02b1cfa015d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:29.855534 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855427    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "f94fe311-68f5-4e17-a22a-02b1cfa015d5" (UID: "f94fe311-68f5-4e17-a22a-02b1cfa015d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:29.855534 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-home" (OuterVolumeSpecName: "home") pod "f94fe311-68f5-4e17-a22a-02b1cfa015d5" (UID: "f94fe311-68f5-4e17-a22a-02b1cfa015d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:29.855678 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855608    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.855678 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855626    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.855678 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.855637    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.857381 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.857357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-dshm" (OuterVolumeSpecName: "dshm") pod "f94fe311-68f5-4e17-a22a-02b1cfa015d5" (UID: "f94fe311-68f5-4e17-a22a-02b1cfa015d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:29.857381 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.857372    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94fe311-68f5-4e17-a22a-02b1cfa015d5-kube-api-access-vnrr7" (OuterVolumeSpecName: "kube-api-access-vnrr7") pod "f94fe311-68f5-4e17-a22a-02b1cfa015d5" (UID: "f94fe311-68f5-4e17-a22a-02b1cfa015d5"). InnerVolumeSpecName "kube-api-access-vnrr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:29.857752 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.857737    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f94fe311-68f5-4e17-a22a-02b1cfa015d5" (UID: "f94fe311-68f5-4e17-a22a-02b1cfa015d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:29.956598 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.956544    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnrr7\" (UniqueName: \"kubernetes.io/projected/f94fe311-68f5-4e17-a22a-02b1cfa015d5-kube-api-access-vnrr7\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.956598 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.956593    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94fe311-68f5-4e17-a22a-02b1cfa015d5-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.956598 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:29.956604    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94fe311-68f5-4e17-a22a-02b1cfa015d5-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:22:30.518663 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.518623    2561 generic.go:356] "Generic (PLEG): container finished" podID="f94fe311-68f5-4e17-a22a-02b1cfa015d5" containerID="93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623" exitCode=0
Sep 16 19:22:30.519082 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.518679    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"
Sep 16 19:22:30.519082 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.518717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt" event={"ID":"f94fe311-68f5-4e17-a22a-02b1cfa015d5","Type":"ContainerDied","Data":"93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623"}
Sep 16 19:22:30.519082 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.518772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt" event={"ID":"f94fe311-68f5-4e17-a22a-02b1cfa015d5","Type":"ContainerDied","Data":"21b0579f5968ad9446861b9eb0149169b9adf341aa524162922840dfcf03a32b"}
Sep 16 19:22:30.519082 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.518795    2561 scope.go:117] "RemoveContainer" containerID="93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623"
Sep 16 19:22:30.536034 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.536011    2561 scope.go:117] "RemoveContainer" containerID="93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623"
Sep 16 19:22:30.536371 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:22:30.536352    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623\": container with ID starting with 93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623 not found: ID does not exist" containerID="93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623"
Sep 16 19:22:30.536419 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.536380    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623"} err="failed to get container status \"93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623\": rpc error: code = NotFound desc = could not find container \"93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623\": container with ID starting with 93958b5aded6be55af6ce96f38bf77723aa5d9b8e8ede9910e50cec0b45d0623 not found: ID does not exist"
Sep 16 19:22:30.546933 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.546903    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"]
Sep 16 19:22:30.553205 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.553182    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997fm6xt"]
Sep 16 19:22:30.749659 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:30.749623    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f94fe311-68f5-4e17-a22a-02b1cfa015d5" path="/var/lib/kubelet/pods/f94fe311-68f5-4e17-a22a-02b1cfa015d5/volumes"
Sep 16 19:22:41.921026 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.920951    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"]
Sep 16 19:22:41.921450 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.921335    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94fe311-68f5-4e17-a22a-02b1cfa015d5" containerName="main"
Sep 16 19:22:41.921450 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.921347    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94fe311-68f5-4e17-a22a-02b1cfa015d5" containerName="main"
Sep 16 19:22:41.921450 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.921418    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f94fe311-68f5-4e17-a22a-02b1cfa015d5" containerName="main"
Sep 16 19:22:41.925899 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.925877    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:41.930008 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.929979    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:22:41.930146 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.930069    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:22:41.930222 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.930178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-2j8l2\""
Sep 16 19:22:41.930483 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.930465    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 16 19:22:41.933640 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.933619    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:22:41.934417 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.934398    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:22:41.969071 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:41.969035    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"]
Sep 16 19:22:42.063151 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.063109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.063151 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.063154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.063393 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.063173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.063393 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.063255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.063393 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.063297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.063393 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.063356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.063393 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.063378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.063608 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.063396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkmz\" (UniqueName: \"kubernetes.io/projected/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kube-api-access-qwkmz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.164659 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.164620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.164659 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.164660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkmz\" (UniqueName: \"kubernetes.io/projected/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kube-api-access-qwkmz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.164915 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.164700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.164915 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.164720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.164915 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.164738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.164915 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.164759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.164915 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.164784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.164915 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.164866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.165193 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.165164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.165245 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.165213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.165293 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.165251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.165505 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.165485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.165551 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.165488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.166942 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.166924    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.167304 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.167284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.179907 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.179850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkmz\" (UniqueName: \"kubernetes.io/projected/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kube-api-access-qwkmz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.236409 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.236372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:42.561669 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.561635    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"]
Sep 16 19:22:42.565051 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:22:42.565025    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820004bc_1024_4cf7_87ec_fcb1cf20d6a4.slice/crio-624331988b07e0a73250aef47be7ccac34dd0d19902afc3ef24940ed8f057845 WatchSource:0}: Error finding container 624331988b07e0a73250aef47be7ccac34dd0d19902afc3ef24940ed8f057845: Status 404 returned error can't find the container with id 624331988b07e0a73250aef47be7ccac34dd0d19902afc3ef24940ed8f057845
Sep 16 19:22:42.572218 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:42.572114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" event={"ID":"820004bc-1024-4cf7-87ec-fcb1cf20d6a4","Type":"ContainerStarted","Data":"624331988b07e0a73250aef47be7ccac34dd0d19902afc3ef24940ed8f057845"}
Sep 16 19:22:43.577997 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:43.577956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" event={"ID":"820004bc-1024-4cf7-87ec-fcb1cf20d6a4","Type":"ContainerStarted","Data":"ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97"}
Sep 16 19:22:45.597121 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:45.597085    2561 generic.go:356] "Generic (PLEG): container finished" podID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerID="ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97" exitCode=0
Sep 16 19:22:45.597506 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:45.597143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" event={"ID":"820004bc-1024-4cf7-87ec-fcb1cf20d6a4","Type":"ContainerDied","Data":"ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97"}
Sep 16 19:22:46.603700 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:46.603663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" event={"ID":"820004bc-1024-4cf7-87ec-fcb1cf20d6a4","Type":"ContainerStarted","Data":"537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e"}
Sep 16 19:22:46.636945 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:46.636880    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podStartSLOduration=3.403704942 podStartE2EDuration="5.636858409s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.233153467s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:46.635624382 +0000 UTC m=+2608.423952745" watchObservedRunningTime="2026-09-16 19:22:46.636858409 +0000 UTC m=+2608.425186776"
Sep 16 19:22:47.237236 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:47.237178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:52.236695 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:52.236655    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:22:52.238354 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:22:52.238319    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:23:02.237822 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:02.237769    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:23:12.237110 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:12.237052    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:23:18.583869 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:18.583830    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"]
Sep 16 19:23:22.237818 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:22.237775    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:23:24.093013 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:24.092974    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" podUID="433ed2ac-cbec-45f4-bef6-9652a87cc291" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:32.237316 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:32.237257    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:23:33.605532 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:33.605491    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" podUID="433ed2ac-cbec-45f4-bef6-9652a87cc291" containerName="main" containerID="cri-o://9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae" gracePeriod=45
Sep 16 19:23:34.093433 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:34.093392    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" podUID="433ed2ac-cbec-45f4-bef6-9652a87cc291" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:42.237443 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:42.237391    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:23:44.092408 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:44.092368    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" podUID="433ed2ac-cbec-45f4-bef6-9652a87cc291" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:44.092881 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:44.092472    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:23:52.237097 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:52.237051    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:23:54.093405 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:23:54.093365    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" podUID="433ed2ac-cbec-45f4-bef6-9652a87cc291" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:24:02.237515 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:02.237472    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:24:03.861289 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.861254    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:24:03.891407 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.891327    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433ed2ac-cbec-45f4-bef6-9652a87cc291-tls-certs\") pod \"433ed2ac-cbec-45f4-bef6-9652a87cc291\" (UID: \"433ed2ac-cbec-45f4-bef6-9652a87cc291\") "
Sep 16 19:24:03.891548 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.891417    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zmpl\" (UniqueName: \"kubernetes.io/projected/433ed2ac-cbec-45f4-bef6-9652a87cc291-kube-api-access-6zmpl\") pod \"433ed2ac-cbec-45f4-bef6-9652a87cc291\" (UID: \"433ed2ac-cbec-45f4-bef6-9652a87cc291\") "
Sep 16 19:24:03.893682 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.893643    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433ed2ac-cbec-45f4-bef6-9652a87cc291-kube-api-access-6zmpl" (OuterVolumeSpecName: "kube-api-access-6zmpl") pod "433ed2ac-cbec-45f4-bef6-9652a87cc291" (UID: "433ed2ac-cbec-45f4-bef6-9652a87cc291"). InnerVolumeSpecName "kube-api-access-6zmpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:03.893682 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.893649    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/433ed2ac-cbec-45f4-bef6-9652a87cc291-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "433ed2ac-cbec-45f4-bef6-9652a87cc291" (UID: "433ed2ac-cbec-45f4-bef6-9652a87cc291"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:03.976749 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.976715    2561 generic.go:356] "Generic (PLEG): container finished" podID="433ed2ac-cbec-45f4-bef6-9652a87cc291" containerID="9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae" exitCode=0
Sep 16 19:24:03.976943 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.976774    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"
Sep 16 19:24:03.976943 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.976800    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" event={"ID":"433ed2ac-cbec-45f4-bef6-9652a87cc291","Type":"ContainerDied","Data":"9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae"}
Sep 16 19:24:03.976943 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.976894    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d" event={"ID":"433ed2ac-cbec-45f4-bef6-9652a87cc291","Type":"ContainerDied","Data":"6893705aae58fde953ddb0590f41a79f0217c16b382152bcbddafcb97fa0fc21"}
Sep 16 19:24:03.976943 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.976925    2561 scope.go:117] "RemoveContainer" containerID="9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae"
Sep 16 19:24:03.986337 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.986320    2561 scope.go:117] "RemoveContainer" containerID="9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae"
Sep 16 19:24:03.986630 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:24:03.986612    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae\": container with ID starting with 9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae not found: ID does not exist" containerID="9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae"
Sep 16 19:24:03.986692 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.986638    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae"} err="failed to get container status \"9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae\": rpc error: code = NotFound desc = could not find container \"9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae\": container with ID starting with 9bed20d5972d33c0f632636a2eb589ac403797d0fcfe8efc579627fd7e0393ae not found: ID does not exist"
Sep 16 19:24:03.992157 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.992134    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433ed2ac-cbec-45f4-bef6-9652a87cc291-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:24:03.992248 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.992159    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zmpl\" (UniqueName: \"kubernetes.io/projected/433ed2ac-cbec-45f4-bef6-9652a87cc291-kube-api-access-6zmpl\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:24:03.998970 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:03.998945    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"]
Sep 16 19:24:04.004112 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:04.004083    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9lq6d"]
Sep 16 19:24:04.750021 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:04.749986    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433ed2ac-cbec-45f4-bef6-9652a87cc291" path="/var/lib/kubelet/pods/433ed2ac-cbec-45f4-bef6-9652a87cc291/volumes"
Sep 16 19:24:12.236998 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:12.236905    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:24:18.881656 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:18.881616    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:24:18.884144 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:18.884118    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:24:20.796411 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.796377    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"]
Sep 16 19:24:20.796917 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.796897    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433ed2ac-cbec-45f4-bef6-9652a87cc291" containerName="main"
Sep 16 19:24:20.796994 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.796921    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="433ed2ac-cbec-45f4-bef6-9652a87cc291" containerName="main"
Sep 16 19:24:20.797053 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.797042    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="433ed2ac-cbec-45f4-bef6-9652a87cc291" containerName="main"
Sep 16 19:24:20.801199 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.801176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:24:20.806236 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.806214    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-6wcwg\""
Sep 16 19:24:20.807253 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.807224    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:24:20.807361 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.807299    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:24:20.807361 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.807335    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:24:20.807482 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.807423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-jmlm9\""
Sep 16 19:24:20.819907 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.819881    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"]
Sep 16 19:24:20.952522 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.952484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzgb2\" (UniqueName: \"kubernetes.io/projected/b9b71180-d0a7-4e62-9984-cd9f05a68e75-kube-api-access-bzgb2\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m\" (UID: \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:24:20.952707 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:20.952540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b71180-d0a7-4e62-9984-cd9f05a68e75-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m\" (UID: \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:24:21.053135 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:21.053044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzgb2\" (UniqueName: \"kubernetes.io/projected/b9b71180-d0a7-4e62-9984-cd9f05a68e75-kube-api-access-bzgb2\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m\" (UID: \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:24:21.053135 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:21.053098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b71180-d0a7-4e62-9984-cd9f05a68e75-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m\" (UID: \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:24:21.055636 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:21.055605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b71180-d0a7-4e62-9984-cd9f05a68e75-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m\" (UID: \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:24:21.062653 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:21.062626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzgb2\" (UniqueName: \"kubernetes.io/projected/b9b71180-d0a7-4e62-9984-cd9f05a68e75-kube-api-access-bzgb2\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m\" (UID: \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:24:21.112044 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:21.112018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:24:21.399951 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:21.399927    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"]
Sep 16 19:24:21.402676 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:24:21.402649    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b71180_d0a7_4e62_9984_cd9f05a68e75.slice/crio-d4ec559ce7ec4e12b602e9f3b62914dc857c312cc7d931897f16cfdc153967c5 WatchSource:0}: Error finding container d4ec559ce7ec4e12b602e9f3b62914dc857c312cc7d931897f16cfdc153967c5: Status 404 returned error can't find the container with id d4ec559ce7ec4e12b602e9f3b62914dc857c312cc7d931897f16cfdc153967c5
Sep 16 19:24:22.066361 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:22.066316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" event={"ID":"b9b71180-d0a7-4e62-9984-cd9f05a68e75","Type":"ContainerStarted","Data":"88c48128c650582fc026f8f26ceaee95668b3f06b84abe516dfa6f13e27110dd"}
Sep 16 19:24:22.066756 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:22.066393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" event={"ID":"b9b71180-d0a7-4e62-9984-cd9f05a68e75","Type":"ContainerStarted","Data":"d4ec559ce7ec4e12b602e9f3b62914dc857c312cc7d931897f16cfdc153967c5"}
Sep 16 19:24:22.066756 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:22.066421    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:24:22.106722 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:22.106662    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" podStartSLOduration=2.106642225 podStartE2EDuration="2.106642225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:22.104884827 +0000 UTC m=+2703.893213190" watchObservedRunningTime="2026-09-16 19:24:22.106642225 +0000 UTC m=+2703.894970592"
Sep 16 19:24:22.237676 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:22.237636    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:24:32.237379 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:32.237326    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:24:42.247304 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:42.247270    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:24:42.255203 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:42.255169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:24:47.682698 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:47.682656    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"]
Sep 16 19:24:53.073807 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:24:53.073774    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:25:02.705811 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:02.705759    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" containerID="cri-o://537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e" gracePeriod=45
Sep 16 19:25:03.248070 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.248026    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:25:03.457746 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.457714    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:25:03.537630 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.537519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tls-certs\") pod \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") "
Sep 16 19:25:03.537630 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.537614    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-model-cache\") pod \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") "
Sep 16 19:25:03.537847 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.537650    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-dshm\") pod \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") "
Sep 16 19:25:03.537847 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.537675    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kserve-provision-location\") pod \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") "
Sep 16 19:25:03.537847 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.537709    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tmp-dir\") pod \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") "
Sep 16 19:25:03.537847 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.537782    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-home\") pod \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") "
Sep 16 19:25:03.537847 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.537809    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-cabundle-cert\") pod \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") "
Sep 16 19:25:03.537847 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.537838    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwkmz\" (UniqueName: \"kubernetes.io/projected/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kube-api-access-qwkmz\") pod \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\" (UID: \"820004bc-1024-4cf7-87ec-fcb1cf20d6a4\") "
Sep 16 19:25:03.538168 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.537947    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "820004bc-1024-4cf7-87ec-fcb1cf20d6a4" (UID: "820004bc-1024-4cf7-87ec-fcb1cf20d6a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:03.538168 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.538119    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.538502 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.538459    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "820004bc-1024-4cf7-87ec-fcb1cf20d6a4" (UID: "820004bc-1024-4cf7-87ec-fcb1cf20d6a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:03.538649 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.538580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-home" (OuterVolumeSpecName: "home") pod "820004bc-1024-4cf7-87ec-fcb1cf20d6a4" (UID: "820004bc-1024-4cf7-87ec-fcb1cf20d6a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:03.540412 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.540379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-dshm" (OuterVolumeSpecName: "dshm") pod "820004bc-1024-4cf7-87ec-fcb1cf20d6a4" (UID: "820004bc-1024-4cf7-87ec-fcb1cf20d6a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:03.540412 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.540391    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "820004bc-1024-4cf7-87ec-fcb1cf20d6a4" (UID: "820004bc-1024-4cf7-87ec-fcb1cf20d6a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:03.540705 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.540505    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kube-api-access-qwkmz" (OuterVolumeSpecName: "kube-api-access-qwkmz") pod "820004bc-1024-4cf7-87ec-fcb1cf20d6a4" (UID: "820004bc-1024-4cf7-87ec-fcb1cf20d6a4"). InnerVolumeSpecName "kube-api-access-qwkmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:03.551086 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.551047    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "820004bc-1024-4cf7-87ec-fcb1cf20d6a4" (UID: "820004bc-1024-4cf7-87ec-fcb1cf20d6a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:03.560011 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.559967    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "820004bc-1024-4cf7-87ec-fcb1cf20d6a4" (UID: "820004bc-1024-4cf7-87ec-fcb1cf20d6a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:03.639348 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.639305    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.639348 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.639343    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.639544 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.639356    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkmz\" (UniqueName: \"kubernetes.io/projected/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kube-api-access-qwkmz\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.639544 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.639370    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.639544 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.639378    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.639544 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.639386    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:25:03.639544 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:03.639395    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820004bc-1024-4cf7-87ec-fcb1cf20d6a4-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:25:04.271773 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.271734    2561 generic.go:356] "Generic (PLEG): container finished" podID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerID="537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e" exitCode=0
Sep 16 19:25:04.272181 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.271786    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"
Sep 16 19:25:04.272181 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.271810    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" event={"ID":"820004bc-1024-4cf7-87ec-fcb1cf20d6a4","Type":"ContainerDied","Data":"537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e"}
Sep 16 19:25:04.272181 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.271860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck" event={"ID":"820004bc-1024-4cf7-87ec-fcb1cf20d6a4","Type":"ContainerDied","Data":"624331988b07e0a73250aef47be7ccac34dd0d19902afc3ef24940ed8f057845"}
Sep 16 19:25:04.272181 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.271885    2561 scope.go:117] "RemoveContainer" containerID="537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e"
Sep 16 19:25:04.280902 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.280884    2561 scope.go:117] "RemoveContainer" containerID="ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97"
Sep 16 19:25:04.288465 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.288444    2561 scope.go:117] "RemoveContainer" containerID="537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e"
Sep 16 19:25:04.288773 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:25:04.288752    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e\": container with ID starting with 537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e not found: ID does not exist" containerID="537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e"
Sep 16 19:25:04.288835 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.288782    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e"} err="failed to get container status \"537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e\": rpc error: code = NotFound desc = could not find container \"537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e\": container with ID starting with 537971f54d863add8d5ecba1ca3c762143dd280774aded7006dd3acb21ca446e not found: ID does not exist"
Sep 16 19:25:04.288835 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.288802    2561 scope.go:117] "RemoveContainer" containerID="ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97"
Sep 16 19:25:04.289023 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:25:04.289005    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97\": container with ID starting with ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97 not found: ID does not exist" containerID="ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97"
Sep 16 19:25:04.289062 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.289030    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97"} err="failed to get container status \"ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97\": rpc error: code = NotFound desc = could not find container \"ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97\": container with ID starting with ab8c5f5642fa2aba0c16513e6d65b43c8b4a3cc2a7dc2d7b31ebdce197946e97 not found: ID does not exist"
Sep 16 19:25:04.308407 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.308375    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"]
Sep 16 19:25:04.312962 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.312937    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-8789f9-94fck"]
Sep 16 19:25:04.752848 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:04.752812    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" path="/var/lib/kubelet/pods/820004bc-1024-4cf7-87ec-fcb1cf20d6a4/volumes"
Sep 16 19:25:40.893932 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.893850    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"]
Sep 16 19:25:40.894375 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.894219    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="storage-initializer"
Sep 16 19:25:40.894375 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.894230    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="storage-initializer"
Sep 16 19:25:40.894375 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.894239    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main"
Sep 16 19:25:40.894375 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.894245    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main"
Sep 16 19:25:40.894375 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.894298    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="820004bc-1024-4cf7-87ec-fcb1cf20d6a4" containerName="main"
Sep 16 19:25:40.897254 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.897236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:25:40.904307 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.904056    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-jv2dq\""
Sep 16 19:25:40.904778 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.904755    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:25:40.904876 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.904783    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:25:40.904876 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.904759    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:25:40.905377 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.905354    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-ck6dc\""
Sep 16 19:25:40.915397 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.915363    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"]
Sep 16 19:25:40.955720 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.955685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nk7f\" (UniqueName: \"kubernetes.io/projected/aebab557-7948-4006-b030-559a6a1b6617-kube-api-access-6nk7f\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks\" (UID: \"aebab557-7948-4006-b030-559a6a1b6617\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:25:40.955884 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:40.955811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aebab557-7948-4006-b030-559a6a1b6617-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks\" (UID: \"aebab557-7948-4006-b030-559a6a1b6617\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:25:41.056562 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:41.056526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nk7f\" (UniqueName: \"kubernetes.io/projected/aebab557-7948-4006-b030-559a6a1b6617-kube-api-access-6nk7f\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks\" (UID: \"aebab557-7948-4006-b030-559a6a1b6617\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:25:41.056772 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:41.056661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aebab557-7948-4006-b030-559a6a1b6617-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks\" (UID: \"aebab557-7948-4006-b030-559a6a1b6617\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:25:41.059169 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:41.059145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aebab557-7948-4006-b030-559a6a1b6617-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks\" (UID: \"aebab557-7948-4006-b030-559a6a1b6617\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:25:41.065114 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:41.065087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nk7f\" (UniqueName: \"kubernetes.io/projected/aebab557-7948-4006-b030-559a6a1b6617-kube-api-access-6nk7f\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks\" (UID: \"aebab557-7948-4006-b030-559a6a1b6617\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:25:41.208822 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:41.208734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:25:41.485999 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:41.485973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"]
Sep 16 19:25:41.488286 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:25:41.488256    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebab557_7948_4006_b030_559a6a1b6617.slice/crio-8470c73a2f5737066a7dd9bb421dbc5b5ac9a9c9c0d1dd49a447af5c854957a9 WatchSource:0}: Error finding container 8470c73a2f5737066a7dd9bb421dbc5b5ac9a9c9c0d1dd49a447af5c854957a9: Status 404 returned error can't find the container with id 8470c73a2f5737066a7dd9bb421dbc5b5ac9a9c9c0d1dd49a447af5c854957a9
Sep 16 19:25:42.444318 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:42.444279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" event={"ID":"aebab557-7948-4006-b030-559a6a1b6617","Type":"ContainerStarted","Data":"b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2"}
Sep 16 19:25:42.444757 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:42.444334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" event={"ID":"aebab557-7948-4006-b030-559a6a1b6617","Type":"ContainerStarted","Data":"8470c73a2f5737066a7dd9bb421dbc5b5ac9a9c9c0d1dd49a447af5c854957a9"}
Sep 16 19:25:42.444757 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:42.444397    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:25:42.463158 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:25:42.463106    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" podStartSLOduration=2.463090106 podStartE2EDuration="2.463090106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:42.461842261 +0000 UTC m=+2784.250170645" watchObservedRunningTime="2026-09-16 19:25:42.463090106 +0000 UTC m=+2784.251418471"
Sep 16 19:26:13.452175 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:26:13.452142    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:26:24.684459 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:26:24.684404    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"]
Sep 16 19:26:33.073764 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:26:33.073724    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" podUID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:39.710125 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:26:39.710076    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" podUID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" containerName="main" containerID="cri-o://88c48128c650582fc026f8f26ceaee95668b3f06b84abe516dfa6f13e27110dd" gracePeriod=45
Sep 16 19:26:43.073835 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:26:43.073786    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" podUID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:53.074135 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:26:53.074091    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" podUID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:53.074622 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:26:53.074259    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:27:03.073833 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:03.073788    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" podUID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:09.838035 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:09.838000    2561 generic.go:356] "Generic (PLEG): container finished" podID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" containerID="88c48128c650582fc026f8f26ceaee95668b3f06b84abe516dfa6f13e27110dd" exitCode=0
Sep 16 19:27:09.838463 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:09.838048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" event={"ID":"b9b71180-d0a7-4e62-9984-cd9f05a68e75","Type":"ContainerDied","Data":"88c48128c650582fc026f8f26ceaee95668b3f06b84abe516dfa6f13e27110dd"}
Sep 16 19:27:09.955694 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:09.955667    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:27:10.038845 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.038808    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzgb2\" (UniqueName: \"kubernetes.io/projected/b9b71180-d0a7-4e62-9984-cd9f05a68e75-kube-api-access-bzgb2\") pod \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\" (UID: \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\") "
Sep 16 19:27:10.039030 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.038870    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b71180-d0a7-4e62-9984-cd9f05a68e75-tls-certs\") pod \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\" (UID: \"b9b71180-d0a7-4e62-9984-cd9f05a68e75\") "
Sep 16 19:27:10.041184 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.041150    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b71180-d0a7-4e62-9984-cd9f05a68e75-kube-api-access-bzgb2" (OuterVolumeSpecName: "kube-api-access-bzgb2") pod "b9b71180-d0a7-4e62-9984-cd9f05a68e75" (UID: "b9b71180-d0a7-4e62-9984-cd9f05a68e75"). InnerVolumeSpecName "kube-api-access-bzgb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:10.041300 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.041250    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b71180-d0a7-4e62-9984-cd9f05a68e75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9b71180-d0a7-4e62-9984-cd9f05a68e75" (UID: "b9b71180-d0a7-4e62-9984-cd9f05a68e75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:10.139396 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.139362    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b71180-d0a7-4e62-9984-cd9f05a68e75-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:27:10.139396 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.139389    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzgb2\" (UniqueName: \"kubernetes.io/projected/b9b71180-d0a7-4e62-9984-cd9f05a68e75-kube-api-access-bzgb2\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:27:10.843830 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.843799    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"
Sep 16 19:27:10.844236 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.843828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m" event={"ID":"b9b71180-d0a7-4e62-9984-cd9f05a68e75","Type":"ContainerDied","Data":"d4ec559ce7ec4e12b602e9f3b62914dc857c312cc7d931897f16cfdc153967c5"}
Sep 16 19:27:10.844236 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.843881    2561 scope.go:117] "RemoveContainer" containerID="88c48128c650582fc026f8f26ceaee95668b3f06b84abe516dfa6f13e27110dd"
Sep 16 19:27:10.860705 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.860681    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"]
Sep 16 19:27:10.864631 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:10.864607    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schehkf4m"]
Sep 16 19:27:12.749026 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:27:12.748985    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" path="/var/lib/kubelet/pods/b9b71180-d0a7-4e62-9984-cd9f05a68e75/volumes"
Sep 16 19:28:15.918902 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:28:15.918857    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"]
Sep 16 19:28:23.451735 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:28:23.451689    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" podUID="aebab557-7948-4006-b030-559a6a1b6617" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:28:30.938754 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:28:30.938709    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" podUID="aebab557-7948-4006-b030-559a6a1b6617" containerName="main" containerID="cri-o://b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2" gracePeriod=45
Sep 16 19:28:33.451841 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:28:33.451794    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" podUID="aebab557-7948-4006-b030-559a6a1b6617" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:28:43.451653 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:28:43.451547    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" podUID="aebab557-7948-4006-b030-559a6a1b6617" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:28:43.452089 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:28:43.451674    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:28:53.451915 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:28:53.451872    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" podUID="aebab557-7948-4006-b030-559a6a1b6617" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:01.187644 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.187615    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:29:01.285745 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.285706    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nk7f\" (UniqueName: \"kubernetes.io/projected/aebab557-7948-4006-b030-559a6a1b6617-kube-api-access-6nk7f\") pod \"aebab557-7948-4006-b030-559a6a1b6617\" (UID: \"aebab557-7948-4006-b030-559a6a1b6617\") "
Sep 16 19:29:01.285950 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.285790    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aebab557-7948-4006-b030-559a6a1b6617-tls-certs\") pod \"aebab557-7948-4006-b030-559a6a1b6617\" (UID: \"aebab557-7948-4006-b030-559a6a1b6617\") "
Sep 16 19:29:01.287995 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.287962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebab557-7948-4006-b030-559a6a1b6617-kube-api-access-6nk7f" (OuterVolumeSpecName: "kube-api-access-6nk7f") pod "aebab557-7948-4006-b030-559a6a1b6617" (UID: "aebab557-7948-4006-b030-559a6a1b6617"). InnerVolumeSpecName "kube-api-access-6nk7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:01.288117 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.288007    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aebab557-7948-4006-b030-559a6a1b6617-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aebab557-7948-4006-b030-559a6a1b6617" (UID: "aebab557-7948-4006-b030-559a6a1b6617"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:01.350860 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.350822    2561 generic.go:356] "Generic (PLEG): container finished" podID="aebab557-7948-4006-b030-559a6a1b6617" containerID="b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2" exitCode=0
Sep 16 19:29:01.351059 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.350870    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"
Sep 16 19:29:01.351059 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.350903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" event={"ID":"aebab557-7948-4006-b030-559a6a1b6617","Type":"ContainerDied","Data":"b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2"}
Sep 16 19:29:01.351059 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.350962    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks" event={"ID":"aebab557-7948-4006-b030-559a6a1b6617","Type":"ContainerDied","Data":"8470c73a2f5737066a7dd9bb421dbc5b5ac9a9c9c0d1dd49a447af5c854957a9"}
Sep 16 19:29:01.351059 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.350985    2561 scope.go:117] "RemoveContainer" containerID="b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2"
Sep 16 19:29:01.359923 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.359905    2561 scope.go:117] "RemoveContainer" containerID="b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2"
Sep 16 19:29:01.360206 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:29:01.360189    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2\": container with ID starting with b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2 not found: ID does not exist" containerID="b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2"
Sep 16 19:29:01.360256 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.360214    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2"} err="failed to get container status \"b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2\": rpc error: code = NotFound desc = could not find container \"b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2\": container with ID starting with b2a3359146d8d3e7c92a86ccb2c29aaa1e2d4359b62e8f34ec88d0da21fadbf2 not found: ID does not exist"
Sep 16 19:29:01.372143 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.372110    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"]
Sep 16 19:29:01.375870 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.375843    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche6ktks"]
Sep 16 19:29:01.386501 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.386479    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nk7f\" (UniqueName: \"kubernetes.io/projected/aebab557-7948-4006-b030-559a6a1b6617-kube-api-access-6nk7f\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.386501 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:01.386499    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aebab557-7948-4006-b030-559a6a1b6617-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:29:02.749410 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:02.749373    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aebab557-7948-4006-b030-559a6a1b6617" path="/var/lib/kubelet/pods/aebab557-7948-4006-b030-559a6a1b6617/volumes"
Sep 16 19:29:13.168770 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.168731    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"]
Sep 16 19:29:13.169212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.169057    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" containerName="main"
Sep 16 19:29:13.169212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.169068    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" containerName="main"
Sep 16 19:29:13.169212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.169081    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aebab557-7948-4006-b030-559a6a1b6617" containerName="main"
Sep 16 19:29:13.169212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.169087    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aebab557-7948-4006-b030-559a6a1b6617" containerName="main"
Sep 16 19:29:13.169212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.169147    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b71180-d0a7-4e62-9984-cd9f05a68e75" containerName="main"
Sep 16 19:29:13.169212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.169159    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aebab557-7948-4006-b030-559a6a1b6617" containerName="main"
Sep 16 19:29:13.173876 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.173858    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.176468 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.176433    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 16 19:29:13.176468 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.176446    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:29:13.176692 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.176474    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 19:29:13.176692 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.176445    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 19:29:13.176692 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.176446    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 19:29:13.176692 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.176582    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-8bt6j\""
Sep 16 19:29:13.185771 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.185745    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"]
Sep 16 19:29:13.290654 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.290615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-model-cache\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.290654 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.290653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4dv\" (UniqueName: \"kubernetes.io/projected/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kube-api-access-lf4dv\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.290883 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.290754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-dshm\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.290883 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.290794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tmp-dir\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.290883 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.290814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tls-certs\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.290883 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.290830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-cabundle-cert\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.290883 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.290855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-home\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.290883 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.290871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.391597 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.391533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-home\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.391597 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.391583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.391873 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.391629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-model-cache\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.391873 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.391648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4dv\" (UniqueName: \"kubernetes.io/projected/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kube-api-access-lf4dv\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.391873 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.391730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-dshm\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.391873 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.391757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tmp-dir\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.391873 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.391775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tls-certs\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.392133 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.391955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-home\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.392133 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.392006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-cabundle-cert\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.392133 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.392012    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-model-cache\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.392133 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.392127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tmp-dir\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.392423 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.392405    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.392547 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.392529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-cabundle-cert\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.394081 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.394060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-dshm\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.394217 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.394199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tls-certs\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.405863 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.405836    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"]
Sep 16 19:29:13.406953 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.406923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4dv\" (UniqueName: \"kubernetes.io/projected/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kube-api-access-lf4dv\") pod \"custom-route-timeout-test-kserve-64dc4b6d4b-d78wx\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.410847 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.410826    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:13.413064 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.413043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-zk44f\""
Sep 16 19:29:13.420638 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.420553    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"]
Sep 16 19:29:13.484265 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.484236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:13.492771 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.492742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16ddc66-4dee-4097-addf-71bd947bc1d5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d\" (UID: \"f16ddc66-4dee-4097-addf-71bd947bc1d5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:13.492889 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.492802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pvc\" (UniqueName: \"kubernetes.io/projected/f16ddc66-4dee-4097-addf-71bd947bc1d5-kube-api-access-k2pvc\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d\" (UID: \"f16ddc66-4dee-4097-addf-71bd947bc1d5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:13.593512 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.593479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16ddc66-4dee-4097-addf-71bd947bc1d5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d\" (UID: \"f16ddc66-4dee-4097-addf-71bd947bc1d5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:13.593676 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.593545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pvc\" (UniqueName: \"kubernetes.io/projected/f16ddc66-4dee-4097-addf-71bd947bc1d5-kube-api-access-k2pvc\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d\" (UID: \"f16ddc66-4dee-4097-addf-71bd947bc1d5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:13.595944 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.595914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16ddc66-4dee-4097-addf-71bd947bc1d5-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d\" (UID: \"f16ddc66-4dee-4097-addf-71bd947bc1d5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:13.601717 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.601698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pvc\" (UniqueName: \"kubernetes.io/projected/f16ddc66-4dee-4097-addf-71bd947bc1d5-kube-api-access-k2pvc\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d\" (UID: \"f16ddc66-4dee-4097-addf-71bd947bc1d5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:13.726509 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.726479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:13.765440 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:13.765404    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"]
Sep 16 19:29:13.769075 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:29:13.769045    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc14f0c_0a39_4c01_86f5_bffb546e76a3.slice/crio-056040b936df6c756d973a13f4fbfdd4d29bb4615c0f45aa9f54eb64aa8a9a97 WatchSource:0}: Error finding container 056040b936df6c756d973a13f4fbfdd4d29bb4615c0f45aa9f54eb64aa8a9a97: Status 404 returned error can't find the container with id 056040b936df6c756d973a13f4fbfdd4d29bb4615c0f45aa9f54eb64aa8a9a97
Sep 16 19:29:14.001116 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:14.001092    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"]
Sep 16 19:29:14.004143 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:29:14.004111    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16ddc66_4dee_4097_addf_71bd947bc1d5.slice/crio-ae2cb85a6f89de4085c9fa3b1c718c0198d3656eaa363cc8a24b5ea1accee378 WatchSource:0}: Error finding container ae2cb85a6f89de4085c9fa3b1c718c0198d3656eaa363cc8a24b5ea1accee378: Status 404 returned error can't find the container with id ae2cb85a6f89de4085c9fa3b1c718c0198d3656eaa363cc8a24b5ea1accee378
Sep 16 19:29:14.411045 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:14.410954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" event={"ID":"f16ddc66-4dee-4097-addf-71bd947bc1d5","Type":"ContainerStarted","Data":"fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab"}
Sep 16 19:29:14.411045 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:14.411018    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:14.411045 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:14.411044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" event={"ID":"f16ddc66-4dee-4097-addf-71bd947bc1d5","Type":"ContainerStarted","Data":"ae2cb85a6f89de4085c9fa3b1c718c0198d3656eaa363cc8a24b5ea1accee378"}
Sep 16 19:29:14.412640 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:14.412614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" event={"ID":"fcc14f0c-0a39-4c01-86f5-bffb546e76a3","Type":"ContainerStarted","Data":"d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be"}
Sep 16 19:29:14.412777 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:14.412655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" event={"ID":"fcc14f0c-0a39-4c01-86f5-bffb546e76a3","Type":"ContainerStarted","Data":"056040b936df6c756d973a13f4fbfdd4d29bb4615c0f45aa9f54eb64aa8a9a97"}
Sep 16 19:29:14.430213 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:14.430162    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" podStartSLOduration=1.430145042 podStartE2EDuration="1.430145042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:14.42886394 +0000 UTC m=+2996.217192306" watchObservedRunningTime="2026-09-16 19:29:14.430145042 +0000 UTC m=+2996.218473408"
Sep 16 19:29:16.422495 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:16.422402    2561 generic.go:356] "Generic (PLEG): container finished" podID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerID="d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be" exitCode=0
Sep 16 19:29:16.422495 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:16.422479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" event={"ID":"fcc14f0c-0a39-4c01-86f5-bffb546e76a3","Type":"ContainerDied","Data":"d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be"}
Sep 16 19:29:17.428958 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:17.428924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" event={"ID":"fcc14f0c-0a39-4c01-86f5-bffb546e76a3","Type":"ContainerStarted","Data":"134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f"}
Sep 16 19:29:17.478949 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:17.478893    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podStartSLOduration=2.191124856 podStartE2EDuration="4.478873753s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.287748897s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:17.47686208 +0000 UTC m=+2999.265190447" watchObservedRunningTime="2026-09-16 19:29:17.478873753 +0000 UTC m=+2999.267202120"
Sep 16 19:29:17.484328 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:17.484297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:18.916062 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:18.916025    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:29:18.918733 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:18.918708    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:29:23.485107 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:23.485068    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:29:23.486517 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:23.486486    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:29:33.485478 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:33.485423    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:29:43.484891 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:43.484839    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:29:45.418495 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:45.418465    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:29:53.485498 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:29:53.485456    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:30:03.485352 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:30:03.485294    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:30:13.485808 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:30:13.485700    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:30:23.485636 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:30:23.485586    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:30:33.484879 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:30:33.484831    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:30:43.485564 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:30:43.485512    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:30:53.484970 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:30:53.484915    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:31:03.485283 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:03.485234    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:31:13.495484 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:13.495442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:31:13.503365 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:13.503340    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:31:18.179909 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:18.179873    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"]
Sep 16 19:31:18.186681 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:18.186646    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"]
Sep 16 19:31:25.418486 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:25.418440    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" podUID="f16ddc66-4dee-4097-addf-71bd947bc1d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:33.204316 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:33.204269    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" containerID="cri-o://134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f" gracePeriod=45
Sep 16 19:31:33.206826 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:33.206793    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" podUID="f16ddc66-4dee-4097-addf-71bd947bc1d5" containerName="main" containerID="cri-o://fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab" gracePeriod=45
Sep 16 19:31:33.496337 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:33.496287    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:31:33.964660 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:33.964628    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:31:34.078553 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.078463    2561 generic.go:356] "Generic (PLEG): container finished" podID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerID="134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f" exitCode=0
Sep 16 19:31:34.078553 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.078516    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"
Sep 16 19:31:34.078553 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.078545    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" event={"ID":"fcc14f0c-0a39-4c01-86f5-bffb546e76a3","Type":"ContainerDied","Data":"134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f"}
Sep 16 19:31:34.078779 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.078614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx" event={"ID":"fcc14f0c-0a39-4c01-86f5-bffb546e76a3","Type":"ContainerDied","Data":"056040b936df6c756d973a13f4fbfdd4d29bb4615c0f45aa9f54eb64aa8a9a97"}
Sep 16 19:31:34.078779 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.078639    2561 scope.go:117] "RemoveContainer" containerID="134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f"
Sep 16 19:31:34.087197 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.087177    2561 scope.go:117] "RemoveContainer" containerID="d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be"
Sep 16 19:31:34.095060 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.095035    2561 scope.go:117] "RemoveContainer" containerID="134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f"
Sep 16 19:31:34.095354 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:31:34.095331    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f\": container with ID starting with 134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f not found: ID does not exist" containerID="134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f"
Sep 16 19:31:34.095433 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.095366    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f"} err="failed to get container status \"134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f\": rpc error: code = NotFound desc = could not find container \"134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f\": container with ID starting with 134c707e45fa253d570394074067a2cd919428dfddb5b6c2ab30669386551e3f not found: ID does not exist"
Sep 16 19:31:34.095433 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.095386    2561 scope.go:117] "RemoveContainer" containerID="d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be"
Sep 16 19:31:34.095680 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:31:34.095662    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be\": container with ID starting with d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be not found: ID does not exist" containerID="d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be"
Sep 16 19:31:34.095734 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.095685    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be"} err="failed to get container status \"d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be\": rpc error: code = NotFound desc = could not find container \"d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be\": container with ID starting with d1e9c32564c026d531b74f456181007d9ed097d8e17cf0df11629075422796be not found: ID does not exist"
Sep 16 19:31:34.131035 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.130991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-home\") pod \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") "
Sep 16 19:31:34.131222 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.131044    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kserve-provision-location\") pod \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") "
Sep 16 19:31:34.131222 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.131066    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf4dv\" (UniqueName: \"kubernetes.io/projected/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kube-api-access-lf4dv\") pod \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") "
Sep 16 19:31:34.131222 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.131097    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tls-certs\") pod \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") "
Sep 16 19:31:34.131222 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.131150    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-cabundle-cert\") pod \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") "
Sep 16 19:31:34.131222 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.131193    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tmp-dir\") pod \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") "
Sep 16 19:31:34.131507 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.131224    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-model-cache\") pod \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") "
Sep 16 19:31:34.131507 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.131254    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-dshm\") pod \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\" (UID: \"fcc14f0c-0a39-4c01-86f5-bffb546e76a3\") "
Sep 16 19:31:34.131657 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.131498    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcc14f0c-0a39-4c01-86f5-bffb546e76a3" (UID: "fcc14f0c-0a39-4c01-86f5-bffb546e76a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:34.131657 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.131562    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fcc14f0c-0a39-4c01-86f5-bffb546e76a3" (UID: "fcc14f0c-0a39-4c01-86f5-bffb546e76a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:34.132029 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.132004    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-home" (OuterVolumeSpecName: "home") pod "fcc14f0c-0a39-4c01-86f5-bffb546e76a3" (UID: "fcc14f0c-0a39-4c01-86f5-bffb546e76a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:34.133755 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.133537    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kube-api-access-lf4dv" (OuterVolumeSpecName: "kube-api-access-lf4dv") pod "fcc14f0c-0a39-4c01-86f5-bffb546e76a3" (UID: "fcc14f0c-0a39-4c01-86f5-bffb546e76a3"). InnerVolumeSpecName "kube-api-access-lf4dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:34.133755 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.133537    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcc14f0c-0a39-4c01-86f5-bffb546e76a3" (UID: "fcc14f0c-0a39-4c01-86f5-bffb546e76a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:34.133932 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.133909    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-dshm" (OuterVolumeSpecName: "dshm") pod "fcc14f0c-0a39-4c01-86f5-bffb546e76a3" (UID: "fcc14f0c-0a39-4c01-86f5-bffb546e76a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:34.143641 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.143612    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcc14f0c-0a39-4c01-86f5-bffb546e76a3" (UID: "fcc14f0c-0a39-4c01-86f5-bffb546e76a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:34.153614 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.153587    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fcc14f0c-0a39-4c01-86f5-bffb546e76a3" (UID: "fcc14f0c-0a39-4c01-86f5-bffb546e76a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:34.232345 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.232306    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:31:34.232345 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.232341    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:31:34.232345 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.232350    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:31:34.232818 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.232357    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:31:34.232818 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.232367    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:31:34.232818 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.232376    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:31:34.232818 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.232384    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf4dv\" (UniqueName: \"kubernetes.io/projected/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-kube-api-access-lf4dv\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:31:34.232818 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.232393    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcc14f0c-0a39-4c01-86f5-bffb546e76a3-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:31:34.403836 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.403804    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"]
Sep 16 19:31:34.414323 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.414290    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-64dc4b6d4b-d78wx"]
Sep 16 19:31:34.749728 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:34.749692    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" path="/var/lib/kubelet/pods/fcc14f0c-0a39-4c01-86f5-bffb546e76a3/volumes"
Sep 16 19:31:35.418442 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:35.418399    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" podUID="f16ddc66-4dee-4097-addf-71bd947bc1d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:45.418852 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:45.418761    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" podUID="f16ddc66-4dee-4097-addf-71bd947bc1d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:45.419213 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:45.418874    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:31:55.419063 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:31:55.419021    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" podUID="f16ddc66-4dee-4097-addf-71bd947bc1d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:03.457416 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:03.457386    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:32:03.565128 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:03.565079    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2pvc\" (UniqueName: \"kubernetes.io/projected/f16ddc66-4dee-4097-addf-71bd947bc1d5-kube-api-access-k2pvc\") pod \"f16ddc66-4dee-4097-addf-71bd947bc1d5\" (UID: \"f16ddc66-4dee-4097-addf-71bd947bc1d5\") "
Sep 16 19:32:03.565342 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:03.565171    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16ddc66-4dee-4097-addf-71bd947bc1d5-tls-certs\") pod \"f16ddc66-4dee-4097-addf-71bd947bc1d5\" (UID: \"f16ddc66-4dee-4097-addf-71bd947bc1d5\") "
Sep 16 19:32:03.567370 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:03.567337    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16ddc66-4dee-4097-addf-71bd947bc1d5-kube-api-access-k2pvc" (OuterVolumeSpecName: "kube-api-access-k2pvc") pod "f16ddc66-4dee-4097-addf-71bd947bc1d5" (UID: "f16ddc66-4dee-4097-addf-71bd947bc1d5"). InnerVolumeSpecName "kube-api-access-k2pvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:03.567479 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:03.567394    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16ddc66-4dee-4097-addf-71bd947bc1d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f16ddc66-4dee-4097-addf-71bd947bc1d5" (UID: "f16ddc66-4dee-4097-addf-71bd947bc1d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:03.666057 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:03.666018    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pvc\" (UniqueName: \"kubernetes.io/projected/f16ddc66-4dee-4097-addf-71bd947bc1d5-kube-api-access-k2pvc\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:32:03.666057 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:03.666051    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16ddc66-4dee-4097-addf-71bd947bc1d5-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:32:04.215781 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.215744    2561 generic.go:356] "Generic (PLEG): container finished" podID="f16ddc66-4dee-4097-addf-71bd947bc1d5" containerID="fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab" exitCode=0
Sep 16 19:32:04.215959 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.215795    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"
Sep 16 19:32:04.215959 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.215822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" event={"ID":"f16ddc66-4dee-4097-addf-71bd947bc1d5","Type":"ContainerDied","Data":"fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab"}
Sep 16 19:32:04.215959 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.215872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d" event={"ID":"f16ddc66-4dee-4097-addf-71bd947bc1d5","Type":"ContainerDied","Data":"ae2cb85a6f89de4085c9fa3b1c718c0198d3656eaa363cc8a24b5ea1accee378"}
Sep 16 19:32:04.215959 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.215893    2561 scope.go:117] "RemoveContainer" containerID="fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab"
Sep 16 19:32:04.224856 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.224827    2561 scope.go:117] "RemoveContainer" containerID="fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab"
Sep 16 19:32:04.225212 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:32:04.225184    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab\": container with ID starting with fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab not found: ID does not exist" containerID="fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab"
Sep 16 19:32:04.225291 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.225222    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab"} err="failed to get container status \"fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab\": rpc error: code = NotFound desc = could not find container \"fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab\": container with ID starting with fcde68fe4d3b20785d98255fdfbedcf280e75e0b51c7e52df9fcb26b8868cfab not found: ID does not exist"
Sep 16 19:32:04.240295 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.240260    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"]
Sep 16 19:32:04.250905 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.250875    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc582x96d"]
Sep 16 19:32:04.749584 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:04.749534    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f16ddc66-4dee-4097-addf-71bd947bc1d5" path="/var/lib/kubelet/pods/f16ddc66-4dee-4097-addf-71bd947bc1d5/volumes"
Sep 16 19:32:20.566982 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.566945    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"]
Sep 16 19:32:20.567360 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.567289    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16ddc66-4dee-4097-addf-71bd947bc1d5" containerName="main"
Sep 16 19:32:20.567360 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.567301    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16ddc66-4dee-4097-addf-71bd947bc1d5" containerName="main"
Sep 16 19:32:20.567360 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.567313    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="storage-initializer"
Sep 16 19:32:20.567360 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.567319    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="storage-initializer"
Sep 16 19:32:20.567360 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.567331    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main"
Sep 16 19:32:20.567360 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.567337    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main"
Sep 16 19:32:20.567557 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.567391    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16ddc66-4dee-4097-addf-71bd947bc1d5" containerName="main"
Sep 16 19:32:20.567557 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.567403    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc14f0c-0a39-4c01-86f5-bffb546e76a3" containerName="main"
Sep 16 19:32:20.571985 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.571963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.574347 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.574319    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:20.574495 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.574325    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:32:20.574495 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.574379    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:32:20.574495 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.574456    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:32:20.575398 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.575371    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:32:20.575678 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.575660    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-d5kwm\""
Sep 16 19:32:20.590006 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.589964    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"]
Sep 16 19:32:20.715736 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.715692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-home\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.715736 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.715742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-model-cache\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.715953 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.715764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tmp-dir\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.715953 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.715786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-dshm\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.715953 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.715845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qsj4\" (UniqueName: \"kubernetes.io/projected/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kube-api-access-9qsj4\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.715953 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.715882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-cabundle-cert\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.715953 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.715948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kserve-provision-location\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.716123 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.715971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tls-certs\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817190 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-cabundle-cert\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817190 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kserve-provision-location\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817410 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tls-certs\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817410 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-home\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817410 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-model-cache\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817410 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tmp-dir\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817670 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-dshm\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817670 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qsj4\" (UniqueName: \"kubernetes.io/projected/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kube-api-access-9qsj4\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817848 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kserve-provision-location\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.817954 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-home\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.818015 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-cabundle-cert\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.818015 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-model-cache\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.818015 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.817993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tmp-dir\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.819901 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.819879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tls-certs\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.820010 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.819917    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-dshm\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.827819 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.827798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qsj4\" (UniqueName: \"kubernetes.io/projected/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kube-api-access-9qsj4\") pod \"router-with-refs-test-kserve-5dccb7c54-xlsdt\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:20.882835 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:20.882796    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:21.182788 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:21.182764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"]
Sep 16 19:32:21.185047 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:32:21.185025    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1c044c_d3f8_49cc_9b6f_71a3b2201bcb.slice/crio-e44eff32afe279ec63d942d61f8b6d35c2a374628abb536ad2d9addbd1c78755 WatchSource:0}: Error finding container e44eff32afe279ec63d942d61f8b6d35c2a374628abb536ad2d9addbd1c78755: Status 404 returned error can't find the container with id e44eff32afe279ec63d942d61f8b6d35c2a374628abb536ad2d9addbd1c78755
Sep 16 19:32:21.296080 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:21.296042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" event={"ID":"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb","Type":"ContainerStarted","Data":"7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456"}
Sep 16 19:32:21.296279 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:21.296095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" event={"ID":"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb","Type":"ContainerStarted","Data":"e44eff32afe279ec63d942d61f8b6d35c2a374628abb536ad2d9addbd1c78755"}
Sep 16 19:32:24.311142 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:24.311107    2561 generic.go:356] "Generic (PLEG): container finished" podID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerID="7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456" exitCode=0
Sep 16 19:32:24.311542 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:24.311180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" event={"ID":"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb","Type":"ContainerDied","Data":"7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456"}
Sep 16 19:32:25.319764 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:25.319723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" event={"ID":"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb","Type":"ContainerStarted","Data":"3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9"}
Sep 16 19:32:25.359209 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:25.359134    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podStartSLOduration=3.116147794 podStartE2EDuration="5.359110079s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.242962285s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:25.356622249 +0000 UTC m=+3187.144950621" watchObservedRunningTime="2026-09-16 19:32:25.359110079 +0000 UTC m=+3187.147438446"
Sep 16 19:32:25.883690 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:25.883644    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:30.883773 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:30.883733    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:32:30.885489 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:30.885460    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:32:40.883669 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:40.883614    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:32:50.883298 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:50.883251    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:32:57.462616 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.462553    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"]
Sep 16 19:32:57.466305 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.466277    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:32:57.470014 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.469984    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:32:57.470193 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.470049    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:32:57.470193 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.470069    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:32:57.470193 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.469990    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-pg5rx\""
Sep 16 19:32:57.470367 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.469990    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-x5sc6\""
Sep 16 19:32:57.475486 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.474974    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"]
Sep 16 19:32:57.558108 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.558061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742tl\" (UniqueName: \"kubernetes.io/projected/62f61691-6453-46fb-a015-56f60c3809ff-kube-api-access-742tl\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9\" (UID: \"62f61691-6453-46fb-a015-56f60c3809ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:32:57.558305 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.558116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f61691-6453-46fb-a015-56f60c3809ff-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9\" (UID: \"62f61691-6453-46fb-a015-56f60c3809ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:32:57.658940 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.658897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742tl\" (UniqueName: \"kubernetes.io/projected/62f61691-6453-46fb-a015-56f60c3809ff-kube-api-access-742tl\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9\" (UID: \"62f61691-6453-46fb-a015-56f60c3809ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:32:57.658940 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.658955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f61691-6453-46fb-a015-56f60c3809ff-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9\" (UID: \"62f61691-6453-46fb-a015-56f60c3809ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:32:57.661592 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.661538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f61691-6453-46fb-a015-56f60c3809ff-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9\" (UID: \"62f61691-6453-46fb-a015-56f60c3809ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:32:57.667015 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.666991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-742tl\" (UniqueName: \"kubernetes.io/projected/62f61691-6453-46fb-a015-56f60c3809ff-kube-api-access-742tl\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9\" (UID: \"62f61691-6453-46fb-a015-56f60c3809ff\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:32:57.779541 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:57.779504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:32:58.068154 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:58.068125    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"]
Sep 16 19:32:58.071312 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:32:58.071279    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f61691_6453_46fb_a015_56f60c3809ff.slice/crio-3fe59f7f21f756e43710ad511d95b1e532f555b4af7179a02d8eb605451a287a WatchSource:0}: Error finding container 3fe59f7f21f756e43710ad511d95b1e532f555b4af7179a02d8eb605451a287a: Status 404 returned error can't find the container with id 3fe59f7f21f756e43710ad511d95b1e532f555b4af7179a02d8eb605451a287a
Sep 16 19:32:58.482386 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:58.482352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" event={"ID":"62f61691-6453-46fb-a015-56f60c3809ff","Type":"ContainerStarted","Data":"2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d"}
Sep 16 19:32:58.482818 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:58.482414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" event={"ID":"62f61691-6453-46fb-a015-56f60c3809ff","Type":"ContainerStarted","Data":"3fe59f7f21f756e43710ad511d95b1e532f555b4af7179a02d8eb605451a287a"}
Sep 16 19:32:58.482818 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:58.482586    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:32:58.519897 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:32:58.519844    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" podStartSLOduration=1.519824656 podStartE2EDuration="1.519824656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:58.519083795 +0000 UTC m=+3220.307412162" watchObservedRunningTime="2026-09-16 19:32:58.519824656 +0000 UTC m=+3220.308153022"
Sep 16 19:33:00.883870 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:33:00.883811    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:33:10.883683 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:33:10.883555    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:33:20.884154 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:33:20.884111    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:33:29.489593 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:33:29.489532    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:33:30.883727 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:33:30.883677    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:33:40.884075 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:33:40.884013    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:33:50.884052 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:33:50.884002    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:34:00.883962 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:00.883914    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:34:10.883585 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:10.883530    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 16 19:34:18.946577 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:18.946527    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:34:18.950430 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:18.950399    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:34:20.893365 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:20.893327    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:34:20.901464 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:20.901435    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:34:26.031793 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:26.031752    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"]
Sep 16 19:34:41.055632 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.055519    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main" containerID="cri-o://3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9" gracePeriod=45
Sep 16 19:34:41.808199 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.808170    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:34:41.903733 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.903641    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-home\") pod \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") "
Sep 16 19:34:41.903733 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.903690    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tmp-dir\") pod \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") "
Sep 16 19:34:41.903733 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.903733    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kserve-provision-location\") pod \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") "
Sep 16 19:34:41.903987 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.903769    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-cabundle-cert\") pod \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") "
Sep 16 19:34:41.903987 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.903828    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-dshm\") pod \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") "
Sep 16 19:34:41.903987 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.903887    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-model-cache\") pod \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") "
Sep 16 19:34:41.903987 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.903912    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qsj4\" (UniqueName: \"kubernetes.io/projected/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kube-api-access-9qsj4\") pod \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") "
Sep 16 19:34:41.903987 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.903946    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tls-certs\") pod \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\" (UID: \"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb\") "
Sep 16 19:34:41.904244 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.904104    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" (UID: "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:41.904244 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.904168    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" (UID: "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:41.904244 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.904230    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:34:41.904438 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.904409    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-home" (OuterVolumeSpecName: "home") pod "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" (UID: "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:41.906300 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.906269    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-dshm" (OuterVolumeSpecName: "dshm") pod "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" (UID: "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:41.906441 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.906354    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kube-api-access-9qsj4" (OuterVolumeSpecName: "kube-api-access-9qsj4") pod "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" (UID: "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb"). InnerVolumeSpecName "kube-api-access-9qsj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:41.906510 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.906447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" (UID: "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:41.916552 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.916524    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" (UID: "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:41.926878 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:41.926846    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" (UID: "fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:42.004754 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.004722    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:34:42.004754 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.004750    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qsj4\" (UniqueName: \"kubernetes.io/projected/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kube-api-access-9qsj4\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:34:42.005001 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.004766    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:34:42.005001 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.004776    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:34:42.005001 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.004784    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:34:42.005001 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.004793    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:34:42.005001 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.004806    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:34:42.005673 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.005646    2561 generic.go:356] "Generic (PLEG): container finished" podID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerID="3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9" exitCode=0
Sep 16 19:34:42.005741 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.005705    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"
Sep 16 19:34:42.005793 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.005731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" event={"ID":"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb","Type":"ContainerDied","Data":"3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9"}
Sep 16 19:34:42.005793 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.005786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt" event={"ID":"fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb","Type":"ContainerDied","Data":"e44eff32afe279ec63d942d61f8b6d35c2a374628abb536ad2d9addbd1c78755"}
Sep 16 19:34:42.005869 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.005816    2561 scope.go:117] "RemoveContainer" containerID="3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9"
Sep 16 19:34:42.015395 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.015374    2561 scope.go:117] "RemoveContainer" containerID="7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456"
Sep 16 19:34:42.023686 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.023663    2561 scope.go:117] "RemoveContainer" containerID="3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9"
Sep 16 19:34:42.024016 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:34:42.023990    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9\": container with ID starting with 3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9 not found: ID does not exist" containerID="3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9"
Sep 16 19:34:42.024123 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.024024    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9"} err="failed to get container status \"3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9\": rpc error: code = NotFound desc = could not find container \"3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9\": container with ID starting with 3b583f223bbc62242cb70c5562e2915d7138ce618c664b398b59a530ced60ae9 not found: ID does not exist"
Sep 16 19:34:42.024123 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.024041    2561 scope.go:117] "RemoveContainer" containerID="7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456"
Sep 16 19:34:42.024341 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:34:42.024319    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456\": container with ID starting with 7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456 not found: ID does not exist" containerID="7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456"
Sep 16 19:34:42.024389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.024346    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456"} err="failed to get container status \"7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456\": rpc error: code = NotFound desc = could not find container \"7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456\": container with ID starting with 7e0a77b0b67a5899b6839027bb378a69ace7b0f86bfb93279c93f74dd94b1456 not found: ID does not exist"
Sep 16 19:34:42.029046 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.029018    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"]
Sep 16 19:34:42.034003 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.033976    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dccb7c54-xlsdt"]
Sep 16 19:34:42.750384 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:42.750348    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" path="/var/lib/kubelet/pods/fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb/volumes"
Sep 16 19:34:59.563755 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:34:59.563719    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"]
Sep 16 19:35:09.489683 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:09.489632    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" podUID="62f61691-6453-46fb-a015-56f60c3809ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:14.585147 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:14.585080    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" podUID="62f61691-6453-46fb-a015-56f60c3809ff" containerName="main" containerID="cri-o://2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d" gracePeriod=45
Sep 16 19:35:19.489146 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:19.489103    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" podUID="62f61691-6453-46fb-a015-56f60c3809ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:29.489699 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:29.489653    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" podUID="62f61691-6453-46fb-a015-56f60c3809ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:29.490213 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:29.489780    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:35:36.342666 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.342626    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"]
Sep 16 19:35:36.343092 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.342978    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main"
Sep 16 19:35:36.343092 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.342992    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main"
Sep 16 19:35:36.343092 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.343005    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="storage-initializer"
Sep 16 19:35:36.343092 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.343010    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="storage-initializer"
Sep 16 19:35:36.343092 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.343072    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa1c044c-d3f8-49cc-9b6f-71a3b2201bcb" containerName="main"
Sep 16 19:35:36.346326 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.346306    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.349925 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.349897    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:35:36.350106 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.349902    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:35:36.350813 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.350790    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:35:36.350963 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.350857    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:35:36.350963 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.350863    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:35:36.350963 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.350946    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-72bgx\""
Sep 16 19:35:36.351192 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.351005    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-h7bfs\""
Sep 16 19:35:36.363084 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.363050    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"]
Sep 16 19:35:36.505905 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.505865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.505905 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.505911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.506140 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.505993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95684169-580f-47a7-9b7d-fac41b9e46fc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.506140 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.506047    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.506140 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.506065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.506140 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.506086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqgdd\" (UniqueName: \"kubernetes.io/projected/95684169-580f-47a7-9b7d-fac41b9e46fc-kube-api-access-mqgdd\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.506140 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.506129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.506321 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.506150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95684169-580f-47a7-9b7d-fac41b9e46fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.607525 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.607437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.607525 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.607473    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.607525 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.607494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqgdd\" (UniqueName: \"kubernetes.io/projected/95684169-580f-47a7-9b7d-fac41b9e46fc-kube-api-access-mqgdd\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.607822 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.607548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.607822 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.607591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95684169-580f-47a7-9b7d-fac41b9e46fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.607822 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.607630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.607822 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.607667    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.607822 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.607711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95684169-580f-47a7-9b7d-fac41b9e46fc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.608076 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.607950    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.608076 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.608014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.608193 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.608106    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.608193 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.608161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.608426 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.608404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95684169-580f-47a7-9b7d-fac41b9e46fc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.610327 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.610295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.610520 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.610502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95684169-580f-47a7-9b7d-fac41b9e46fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.621265 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.621225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqgdd\" (UniqueName: \"kubernetes.io/projected/95684169-580f-47a7-9b7d-fac41b9e46fc-kube-api-access-mqgdd\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.657406 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.657365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:36.951268 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:36.951240    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"]
Sep 16 19:35:36.952982 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:35:36.952948    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95684169_580f_47a7_9b7d_fac41b9e46fc.slice/crio-1cb378599fa8e8a4667460e6de971ccc942b7e7451d88a2661a2fcbbe7c39293 WatchSource:0}: Error finding container 1cb378599fa8e8a4667460e6de971ccc942b7e7451d88a2661a2fcbbe7c39293: Status 404 returned error can't find the container with id 1cb378599fa8e8a4667460e6de971ccc942b7e7451d88a2661a2fcbbe7c39293
Sep 16 19:35:37.252000 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:37.251954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" event={"ID":"95684169-580f-47a7-9b7d-fac41b9e46fc","Type":"ContainerStarted","Data":"34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31"}
Sep 16 19:35:37.252193 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:37.252012    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:37.252193 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:37.252033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" event={"ID":"95684169-580f-47a7-9b7d-fac41b9e46fc","Type":"ContainerStarted","Data":"1cb378599fa8e8a4667460e6de971ccc942b7e7451d88a2661a2fcbbe7c39293"}
Sep 16 19:35:38.261318 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:38.261273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" event={"ID":"95684169-580f-47a7-9b7d-fac41b9e46fc","Type":"ContainerStarted","Data":"63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096"}
Sep 16 19:35:39.489197 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:39.489158    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" podUID="62f61691-6453-46fb-a015-56f60c3809ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:40.273140 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:40.273097    2561 generic.go:356] "Generic (PLEG): container finished" podID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerID="63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096" exitCode=0
Sep 16 19:35:40.273140 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:40.273143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" event={"ID":"95684169-580f-47a7-9b7d-fac41b9e46fc","Type":"ContainerDied","Data":"63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096"}
Sep 16 19:35:41.280784 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:41.280744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" event={"ID":"95684169-580f-47a7-9b7d-fac41b9e46fc","Type":"ContainerStarted","Data":"2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727"}
Sep 16 19:35:41.309856 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:41.309796    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podStartSLOduration=3.015766265 podStartE2EDuration="5.309775714s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.294009449s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:41.30796421 +0000 UTC m=+3383.096292600" watchObservedRunningTime="2026-09-16 19:35:41.309775714 +0000 UTC m=+3383.098104079"
Sep 16 19:35:44.836805 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:44.836774    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:35:44.986881 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:44.986847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-742tl\" (UniqueName: \"kubernetes.io/projected/62f61691-6453-46fb-a015-56f60c3809ff-kube-api-access-742tl\") pod \"62f61691-6453-46fb-a015-56f60c3809ff\" (UID: \"62f61691-6453-46fb-a015-56f60c3809ff\") "
Sep 16 19:35:44.987058 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:44.986936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f61691-6453-46fb-a015-56f60c3809ff-tls-certs\") pod \"62f61691-6453-46fb-a015-56f60c3809ff\" (UID: \"62f61691-6453-46fb-a015-56f60c3809ff\") "
Sep 16 19:35:44.989284 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:44.989237    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f61691-6453-46fb-a015-56f60c3809ff-kube-api-access-742tl" (OuterVolumeSpecName: "kube-api-access-742tl") pod "62f61691-6453-46fb-a015-56f60c3809ff" (UID: "62f61691-6453-46fb-a015-56f60c3809ff"). InnerVolumeSpecName "kube-api-access-742tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:44.989284 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:44.989256    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f61691-6453-46fb-a015-56f60c3809ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62f61691-6453-46fb-a015-56f60c3809ff" (UID: "62f61691-6453-46fb-a015-56f60c3809ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:45.088209 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.088174    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f61691-6453-46fb-a015-56f60c3809ff-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:35:45.088209 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.088204    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-742tl\" (UniqueName: \"kubernetes.io/projected/62f61691-6453-46fb-a015-56f60c3809ff-kube-api-access-742tl\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:35:45.304066 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.303966    2561 generic.go:356] "Generic (PLEG): container finished" podID="62f61691-6453-46fb-a015-56f60c3809ff" containerID="2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d" exitCode=0
Sep 16 19:35:45.304066 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.304011    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"
Sep 16 19:35:45.304066 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.304049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" event={"ID":"62f61691-6453-46fb-a015-56f60c3809ff","Type":"ContainerDied","Data":"2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d"}
Sep 16 19:35:45.304357 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.304104    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9" event={"ID":"62f61691-6453-46fb-a015-56f60c3809ff","Type":"ContainerDied","Data":"3fe59f7f21f756e43710ad511d95b1e532f555b4af7179a02d8eb605451a287a"}
Sep 16 19:35:45.304357 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.304129    2561 scope.go:117] "RemoveContainer" containerID="2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d"
Sep 16 19:35:45.314272 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.314251    2561 scope.go:117] "RemoveContainer" containerID="2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d"
Sep 16 19:35:45.314581 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:35:45.314549    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d\": container with ID starting with 2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d not found: ID does not exist" containerID="2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d"
Sep 16 19:35:45.314666 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.314589    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d"} err="failed to get container status \"2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d\": rpc error: code = NotFound desc = could not find container \"2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d\": container with ID starting with 2f2ca92d5669687792c117b3e2d44e81a03def4b9d17f281e7a3c23e2a111d2d not found: ID does not exist"
Sep 16 19:35:45.326107 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.326047    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"]
Sep 16 19:35:45.327958 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:45.327927    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-888z9"]
Sep 16 19:35:46.657632 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:46.657594    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:46.658181 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:46.657663    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:35:46.658751 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:46.658717    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:35:46.751168 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:46.751124    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f61691-6453-46fb-a015-56f60c3809ff" path="/var/lib/kubelet/pods/62f61691-6453-46fb-a015-56f60c3809ff/volumes"
Sep 16 19:35:56.657682 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:56.657633    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:35:56.677310 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:35:56.677276    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:36:06.657730 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:36:06.657675    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:36:16.658795 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:36:16.658677    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:36:26.657759 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:36:26.657696    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:36:36.658489 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:36:36.658438    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:36:46.657976 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:36:46.657925    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:36:56.657723 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:36:56.657677    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:37:06.657794 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:37:06.657748    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:37:16.657834 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:37:16.657775    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:37:26.657847 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:37:26.657782    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8001/health\": dial tcp 10.132.0.58:8001: connect: connection refused"
Sep 16 19:37:36.667672 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:37:36.667636    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:37:36.686783 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:37:36.686745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:38:25.957895 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:25.957857    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"]
Sep 16 19:38:40.981038 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:40.980985    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main" containerID="cri-o://2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727" gracePeriod=45
Sep 16 19:38:41.597579 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.597535    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="llm-d-routing-sidecar" containerID="cri-o://34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31" gracePeriod=45
Sep 16 19:38:41.846356 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.846325    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:38:41.940253 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940210    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-kserve-provision-location\") pod \"95684169-580f-47a7-9b7d-fac41b9e46fc\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") "
Sep 16 19:38:41.940253 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95684169-580f-47a7-9b7d-fac41b9e46fc-tls-certs\") pod \"95684169-580f-47a7-9b7d-fac41b9e46fc\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") "
Sep 16 19:38:41.940499 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940293    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-tmp-dir\") pod \"95684169-580f-47a7-9b7d-fac41b9e46fc\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") "
Sep 16 19:38:41.940499 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940324    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-home\") pod \"95684169-580f-47a7-9b7d-fac41b9e46fc\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") "
Sep 16 19:38:41.940499 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940361    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqgdd\" (UniqueName: \"kubernetes.io/projected/95684169-580f-47a7-9b7d-fac41b9e46fc-kube-api-access-mqgdd\") pod \"95684169-580f-47a7-9b7d-fac41b9e46fc\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") "
Sep 16 19:38:41.940499 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940403    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-dshm\") pod \"95684169-580f-47a7-9b7d-fac41b9e46fc\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") "
Sep 16 19:38:41.940499 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95684169-580f-47a7-9b7d-fac41b9e46fc-cabundle-cert\") pod \"95684169-580f-47a7-9b7d-fac41b9e46fc\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") "
Sep 16 19:38:41.940499 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940486    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-model-cache\") pod \"95684169-580f-47a7-9b7d-fac41b9e46fc\" (UID: \"95684169-580f-47a7-9b7d-fac41b9e46fc\") "
Sep 16 19:38:41.940898 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940871    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "95684169-580f-47a7-9b7d-fac41b9e46fc" (UID: "95684169-580f-47a7-9b7d-fac41b9e46fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:41.940980 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.940949    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95684169-580f-47a7-9b7d-fac41b9e46fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "95684169-580f-47a7-9b7d-fac41b9e46fc" (UID: "95684169-580f-47a7-9b7d-fac41b9e46fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:38:41.941219 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.941173    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-home" (OuterVolumeSpecName: "home") pod "95684169-580f-47a7-9b7d-fac41b9e46fc" (UID: "95684169-580f-47a7-9b7d-fac41b9e46fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:41.943074 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.943038    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-dshm" (OuterVolumeSpecName: "dshm") pod "95684169-580f-47a7-9b7d-fac41b9e46fc" (UID: "95684169-580f-47a7-9b7d-fac41b9e46fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:41.943414 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.943390    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95684169-580f-47a7-9b7d-fac41b9e46fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95684169-580f-47a7-9b7d-fac41b9e46fc" (UID: "95684169-580f-47a7-9b7d-fac41b9e46fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:41.943517 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.943409    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95684169-580f-47a7-9b7d-fac41b9e46fc-kube-api-access-mqgdd" (OuterVolumeSpecName: "kube-api-access-mqgdd") pod "95684169-580f-47a7-9b7d-fac41b9e46fc" (UID: "95684169-580f-47a7-9b7d-fac41b9e46fc"). InnerVolumeSpecName "kube-api-access-mqgdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:41.954006 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.953979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95684169-580f-47a7-9b7d-fac41b9e46fc" (UID: "95684169-580f-47a7-9b7d-fac41b9e46fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:41.964486 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:41.964459    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95684169-580f-47a7-9b7d-fac41b9e46fc" (UID: "95684169-580f-47a7-9b7d-fac41b9e46fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:42.042024 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.041981    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:38:42.042024 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.042018    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:38:42.042024 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.042032    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqgdd\" (UniqueName: \"kubernetes.io/projected/95684169-580f-47a7-9b7d-fac41b9e46fc-kube-api-access-mqgdd\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:38:42.042511 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.042042    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:38:42.042511 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.042051    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95684169-580f-47a7-9b7d-fac41b9e46fc-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:38:42.042511 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.042058    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:38:42.042511 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.042066    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95684169-580f-47a7-9b7d-fac41b9e46fc-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:38:42.042511 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.042074    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95684169-580f-47a7-9b7d-fac41b9e46fc-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:38:42.151716 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.151617    2561 generic.go:356] "Generic (PLEG): container finished" podID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerID="2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727" exitCode=0
Sep 16 19:38:42.151716 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.151643    2561 generic.go:356] "Generic (PLEG): container finished" podID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerID="34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31" exitCode=0
Sep 16 19:38:42.151716 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.151680    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"
Sep 16 19:38:42.151716 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.151702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" event={"ID":"95684169-580f-47a7-9b7d-fac41b9e46fc","Type":"ContainerDied","Data":"2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727"}
Sep 16 19:38:42.152031 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.151770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" event={"ID":"95684169-580f-47a7-9b7d-fac41b9e46fc","Type":"ContainerDied","Data":"34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31"}
Sep 16 19:38:42.152031 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.151788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7" event={"ID":"95684169-580f-47a7-9b7d-fac41b9e46fc","Type":"ContainerDied","Data":"1cb378599fa8e8a4667460e6de971ccc942b7e7451d88a2661a2fcbbe7c39293"}
Sep 16 19:38:42.152031 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.151810    2561 scope.go:117] "RemoveContainer" containerID="2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727"
Sep 16 19:38:42.160710 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.160691    2561 scope.go:117] "RemoveContainer" containerID="63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096"
Sep 16 19:38:42.168559 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.168542    2561 scope.go:117] "RemoveContainer" containerID="34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31"
Sep 16 19:38:42.176297 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.176274    2561 scope.go:117] "RemoveContainer" containerID="2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727"
Sep 16 19:38:42.176911 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:38:42.176878    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727\": container with ID starting with 2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727 not found: ID does not exist" containerID="2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727"
Sep 16 19:38:42.176998 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.176919    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727"} err="failed to get container status \"2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727\": rpc error: code = NotFound desc = could not find container \"2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727\": container with ID starting with 2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727 not found: ID does not exist"
Sep 16 19:38:42.176998 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.176946    2561 scope.go:117] "RemoveContainer" containerID="63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096"
Sep 16 19:38:42.177230 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:38:42.177214    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096\": container with ID starting with 63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096 not found: ID does not exist" containerID="63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096"
Sep 16 19:38:42.177275 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.177234    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096"} err="failed to get container status \"63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096\": rpc error: code = NotFound desc = could not find container \"63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096\": container with ID starting with 63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096 not found: ID does not exist"
Sep 16 19:38:42.177275 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.177249    2561 scope.go:117] "RemoveContainer" containerID="34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31"
Sep 16 19:38:42.177537 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:38:42.177513    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31\": container with ID starting with 34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31 not found: ID does not exist" containerID="34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31"
Sep 16 19:38:42.177609 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.177544    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31"} err="failed to get container status \"34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31\": rpc error: code = NotFound desc = could not find container \"34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31\": container with ID starting with 34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31 not found: ID does not exist"
Sep 16 19:38:42.177609 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.177590    2561 scope.go:117] "RemoveContainer" containerID="2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727"
Sep 16 19:38:42.177880 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.177862    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727"} err="failed to get container status \"2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727\": rpc error: code = NotFound desc = could not find container \"2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727\": container with ID starting with 2e4639845b0d75a8c1efb78bf93786ea367545a7dd7483258948bf74905f0727 not found: ID does not exist"
Sep 16 19:38:42.177926 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.177882    2561 scope.go:117] "RemoveContainer" containerID="63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096"
Sep 16 19:38:42.177976 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.177923    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"]
Sep 16 19:38:42.178139 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.178121    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096"} err="failed to get container status \"63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096\": rpc error: code = NotFound desc = could not find container \"63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096\": container with ID starting with 63d68d236a9f38f3b554305b156e9d5f316af8030a8ebd219c2366ebf2991096 not found: ID does not exist"
Sep 16 19:38:42.178204 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.178139    2561 scope.go:117] "RemoveContainer" containerID="34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31"
Sep 16 19:38:42.178336 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.178325    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31"} err="failed to get container status \"34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31\": rpc error: code = NotFound desc = could not find container \"34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31\": container with ID starting with 34812d9cd7e0b8c5d0edd93a41a42c370f6ab8e0bfcaaf318fd21f379c948e31 not found: ID does not exist"
Sep 16 19:38:42.182042 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.182019    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-598b4bd6f-6ftr7"]
Sep 16 19:38:42.753839 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:38:42.750342    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" path="/var/lib/kubelet/pods/95684169-580f-47a7-9b7d-fac41b9e46fc/volumes"
Sep 16 19:39:18.977372 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:18.977338    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:39:18.983044 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:18.983016    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:39:37.132970 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.132934    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"]
Sep 16 19:39:37.133389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133266    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="llm-d-routing-sidecar"
Sep 16 19:39:37.133389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133278    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="llm-d-routing-sidecar"
Sep 16 19:39:37.133389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133303    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f61691-6453-46fb-a015-56f60c3809ff" containerName="main"
Sep 16 19:39:37.133389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133310    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f61691-6453-46fb-a015-56f60c3809ff" containerName="main"
Sep 16 19:39:37.133389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133320    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="storage-initializer"
Sep 16 19:39:37.133389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133327    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="storage-initializer"
Sep 16 19:39:37.133389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133343    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main"
Sep 16 19:39:37.133389 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133348    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main"
Sep 16 19:39:37.133706 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133412    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="llm-d-routing-sidecar"
Sep 16 19:39:37.133706 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133423    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f61691-6453-46fb-a015-56f60c3809ff" containerName="main"
Sep 16 19:39:37.133706 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.133429    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="95684169-580f-47a7-9b7d-fac41b9e46fc" containerName="main"
Sep 16 19:39:37.136674 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.136654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.140446 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.140424    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:39:37.140446 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.140442    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:39:37.141600 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.141553    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-jpvfk\""
Sep 16 19:39:37.144050 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.144022    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:39:37.144168 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.144049    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:39:37.144168 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.144135    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-cnqpm\""
Sep 16 19:39:37.144942 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.144924    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:39:37.161388 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.161358    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"]
Sep 16 19:39:37.236678 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.236630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.236861 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.236724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.236861 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.236747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.236861 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.236795    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz8jj\" (UniqueName: \"kubernetes.io/projected/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kube-api-access-mz8jj\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.236861 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.236851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-home\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.237037 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.236874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-dshm\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.237037 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.236911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565adc5d-b9ed-457c-a2e3-dedd7796d92a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.237037 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.236967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.337649 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.337615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.337848 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.337673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.337848 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.337804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.337848 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.337841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz8jj\" (UniqueName: \"kubernetes.io/projected/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kube-api-access-mz8jj\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.338017 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.337956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-home\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.338017 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.337984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-dshm\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.338114 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.338024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565adc5d-b9ed-457c-a2e3-dedd7796d92a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.338114 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.338033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.338114 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.338106    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.338276 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.338130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.338276 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.338143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.338276 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.338232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-home\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.338643 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.338621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565adc5d-b9ed-457c-a2e3-dedd7796d92a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.340235 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.340205    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-dshm\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.340500 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.340483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.357545 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.357504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz8jj\" (UniqueName: \"kubernetes.io/projected/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kube-api-access-mz8jj\") pod \"custom-route-timeout-pd-test-kserve-7b845d6478-ppb49\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.447310 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.447217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:37.596257 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.596220    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"]
Sep 16 19:39:37.600396 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.600372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:39:37.602753 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.602730    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-8w6rs\""
Sep 16 19:39:37.612148 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.612121    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"]
Sep 16 19:39:37.741081 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.741045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hj5j\" (UniqueName: \"kubernetes.io/projected/a2d62190-33b7-4886-a4ed-708ef3a0559a-kube-api-access-7hj5j\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6\" (UID: \"a2d62190-33b7-4886-a4ed-708ef3a0559a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:39:37.741280 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.741132    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d62190-33b7-4886-a4ed-708ef3a0559a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6\" (UID: \"a2d62190-33b7-4886-a4ed-708ef3a0559a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:39:37.749058 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.749031    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"]
Sep 16 19:39:37.751177 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:39:37.751151    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565adc5d_b9ed_457c_a2e3_dedd7796d92a.slice/crio-a48c053293830e75ffabb27b133e77eb70140cefc3563f368299b1050f11201c WatchSource:0}: Error finding container a48c053293830e75ffabb27b133e77eb70140cefc3563f368299b1050f11201c: Status 404 returned error can't find the container with id a48c053293830e75ffabb27b133e77eb70140cefc3563f368299b1050f11201c
Sep 16 19:39:37.841927 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.841889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hj5j\" (UniqueName: \"kubernetes.io/projected/a2d62190-33b7-4886-a4ed-708ef3a0559a-kube-api-access-7hj5j\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6\" (UID: \"a2d62190-33b7-4886-a4ed-708ef3a0559a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:39:37.842119 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.842013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d62190-33b7-4886-a4ed-708ef3a0559a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6\" (UID: \"a2d62190-33b7-4886-a4ed-708ef3a0559a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:39:37.845114 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.845091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d62190-33b7-4886-a4ed-708ef3a0559a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6\" (UID: \"a2d62190-33b7-4886-a4ed-708ef3a0559a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:39:37.863322 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.863294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hj5j\" (UniqueName: \"kubernetes.io/projected/a2d62190-33b7-4886-a4ed-708ef3a0559a-kube-api-access-7hj5j\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6\" (UID: \"a2d62190-33b7-4886-a4ed-708ef3a0559a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:39:37.912284 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:37.912249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:39:38.194637 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:38.194606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"]
Sep 16 19:39:38.195265 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:39:38.195233    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d62190_33b7_4886_a4ed_708ef3a0559a.slice/crio-12af725dbc6a85db3e69a3b036960f6b2bf8795d6e0b6334741561edbbce59b7 WatchSource:0}: Error finding container 12af725dbc6a85db3e69a3b036960f6b2bf8795d6e0b6334741561edbbce59b7: Status 404 returned error can't find the container with id 12af725dbc6a85db3e69a3b036960f6b2bf8795d6e0b6334741561edbbce59b7
Sep 16 19:39:38.416983 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:38.416935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" event={"ID":"a2d62190-33b7-4886-a4ed-708ef3a0559a","Type":"ContainerStarted","Data":"c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991"}
Sep 16 19:39:38.417170 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:38.417001    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" event={"ID":"a2d62190-33b7-4886-a4ed-708ef3a0559a","Type":"ContainerStarted","Data":"12af725dbc6a85db3e69a3b036960f6b2bf8795d6e0b6334741561edbbce59b7"}
Sep 16 19:39:38.417170 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:38.417124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:39:38.418686 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:38.418660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" event={"ID":"565adc5d-b9ed-457c-a2e3-dedd7796d92a","Type":"ContainerStarted","Data":"90bc41a3c7f8059bc99b171d904596250f322bf2e99de7484d8fefef0b606cbc"}
Sep 16 19:39:38.418824 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:38.418699    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" event={"ID":"565adc5d-b9ed-457c-a2e3-dedd7796d92a","Type":"ContainerStarted","Data":"a48c053293830e75ffabb27b133e77eb70140cefc3563f368299b1050f11201c"}
Sep 16 19:39:38.418824 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:38.418771    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:38.439234 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:38.439137    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" podStartSLOduration=1.439114553 podStartE2EDuration="1.439114553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:38.438806321 +0000 UTC m=+3620.227134688" watchObservedRunningTime="2026-09-16 19:39:38.439114553 +0000 UTC m=+3620.227442919"
Sep 16 19:39:39.425407 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:39.425363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" event={"ID":"565adc5d-b9ed-457c-a2e3-dedd7796d92a","Type":"ContainerStarted","Data":"c0eaf5d94197403c110fcf2db0313f9ddebe040f35c7f7b1b35068e4a368cc53"}
Sep 16 19:39:41.436451 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:41.436416    2561 generic.go:356] "Generic (PLEG): container finished" podID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerID="c0eaf5d94197403c110fcf2db0313f9ddebe040f35c7f7b1b35068e4a368cc53" exitCode=0
Sep 16 19:39:41.436870 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:41.436493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" event={"ID":"565adc5d-b9ed-457c-a2e3-dedd7796d92a","Type":"ContainerDied","Data":"c0eaf5d94197403c110fcf2db0313f9ddebe040f35c7f7b1b35068e4a368cc53"}
Sep 16 19:39:42.443548 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:42.443512    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" event={"ID":"565adc5d-b9ed-457c-a2e3-dedd7796d92a","Type":"ContainerStarted","Data":"b1959521bda91509f37abdb32df6a10a58bb98a554f4d143b2d7ac12c81c7182"}
Sep 16 19:39:42.470526 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:42.470462    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podStartSLOduration=3.223159018 podStartE2EDuration="5.470438892s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.247279874s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:42.4683435 +0000 UTC m=+3624.256671870" watchObservedRunningTime="2026-09-16 19:39:42.470438892 +0000 UTC m=+3624.258767258"
Sep 16 19:39:47.447971 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:47.447924    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:47.448479 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:47.448006    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:47.449305 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:47.449267    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:39:47.462223 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:47.462194    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:39:57.447661 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:39:57.447608    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:40:07.447855 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:40:07.447795    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:40:09.427233 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:40:09.427199    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:40:17.447742 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:40:17.447689    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:40:27.447845 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:40:27.447792    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:40:37.448028 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:40:37.447987    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:40:47.448276 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:40:47.448169    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:40:57.448178 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:40:57.448122    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:41:07.447692 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:41:07.447633    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:41:17.448402 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:41:17.448325    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:41:27.448543 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:41:27.448482    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8001/health\": dial tcp 10.132.0.59:8001: connect: connection refused"
Sep 16 19:41:37.457793 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:41:37.457754    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:41:37.476181 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:41:37.476155    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:42:22.495484 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:22.495394    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"]
Sep 16 19:42:22.509117 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:22.509079    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"]
Sep 16 19:42:29.427769 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:29.427732    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" podUID="a2d62190-33b7-4886-a4ed-708ef3a0559a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:37.516776 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:37.516707    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" podUID="a2d62190-33b7-4886-a4ed-708ef3a0559a" containerName="main" containerID="cri-o://c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991" gracePeriod=45
Sep 16 19:42:37.531669 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:37.531628    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main" containerID="cri-o://b1959521bda91509f37abdb32df6a10a58bb98a554f4d143b2d7ac12c81c7182" gracePeriod=45
Sep 16 19:42:38.148954 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.148915    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="llm-d-routing-sidecar" containerID="cri-o://90bc41a3c7f8059bc99b171d904596250f322bf2e99de7484d8fefef0b606cbc" gracePeriod=45
Sep 16 19:42:38.289821 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.289773    2561 generic.go:356] "Generic (PLEG): container finished" podID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerID="b1959521bda91509f37abdb32df6a10a58bb98a554f4d143b2d7ac12c81c7182" exitCode=0
Sep 16 19:42:38.289821 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.289808    2561 generic.go:356] "Generic (PLEG): container finished" podID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerID="90bc41a3c7f8059bc99b171d904596250f322bf2e99de7484d8fefef0b606cbc" exitCode=0
Sep 16 19:42:38.290034 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.289848    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" event={"ID":"565adc5d-b9ed-457c-a2e3-dedd7796d92a","Type":"ContainerDied","Data":"b1959521bda91509f37abdb32df6a10a58bb98a554f4d143b2d7ac12c81c7182"}
Sep 16 19:42:38.290034 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.289906    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" event={"ID":"565adc5d-b9ed-457c-a2e3-dedd7796d92a","Type":"ContainerDied","Data":"90bc41a3c7f8059bc99b171d904596250f322bf2e99de7484d8fefef0b606cbc"}
Sep 16 19:42:38.400192 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.400163    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:42:38.475697 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.475653    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-home\") pod \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") "
Sep 16 19:42:38.475907 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.475713    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tmp-dir\") pod \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") "
Sep 16 19:42:38.475907 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.475763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-model-cache\") pod \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") "
Sep 16 19:42:38.475907 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.475801    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tls-certs\") pod \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") "
Sep 16 19:42:38.475907 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.475851    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz8jj\" (UniqueName: \"kubernetes.io/projected/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kube-api-access-mz8jj\") pod \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") "
Sep 16 19:42:38.475907 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.475879    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kserve-provision-location\") pod \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") "
Sep 16 19:42:38.476183 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.475934    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-dshm\") pod \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") "
Sep 16 19:42:38.476183 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.475959    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565adc5d-b9ed-457c-a2e3-dedd7796d92a-cabundle-cert\") pod \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\" (UID: \"565adc5d-b9ed-457c-a2e3-dedd7796d92a\") "
Sep 16 19:42:38.476183 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.476031    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-model-cache" (OuterVolumeSpecName: "model-cache") pod "565adc5d-b9ed-457c-a2e3-dedd7796d92a" (UID: "565adc5d-b9ed-457c-a2e3-dedd7796d92a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.476350 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.476275    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.476350 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.476307    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-home" (OuterVolumeSpecName: "home") pod "565adc5d-b9ed-457c-a2e3-dedd7796d92a" (UID: "565adc5d-b9ed-457c-a2e3-dedd7796d92a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.476457 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.476433    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565adc5d-b9ed-457c-a2e3-dedd7796d92a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "565adc5d-b9ed-457c-a2e3-dedd7796d92a" (UID: "565adc5d-b9ed-457c-a2e3-dedd7796d92a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:42:38.478477 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.478437    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-dshm" (OuterVolumeSpecName: "dshm") pod "565adc5d-b9ed-457c-a2e3-dedd7796d92a" (UID: "565adc5d-b9ed-457c-a2e3-dedd7796d92a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.478653 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.478479    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kube-api-access-mz8jj" (OuterVolumeSpecName: "kube-api-access-mz8jj") pod "565adc5d-b9ed-457c-a2e3-dedd7796d92a" (UID: "565adc5d-b9ed-457c-a2e3-dedd7796d92a"). InnerVolumeSpecName "kube-api-access-mz8jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:38.478653 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.478488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "565adc5d-b9ed-457c-a2e3-dedd7796d92a" (UID: "565adc5d-b9ed-457c-a2e3-dedd7796d92a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:38.488538 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.488504    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "565adc5d-b9ed-457c-a2e3-dedd7796d92a" (UID: "565adc5d-b9ed-457c-a2e3-dedd7796d92a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.500751 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.500706    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "565adc5d-b9ed-457c-a2e3-dedd7796d92a" (UID: "565adc5d-b9ed-457c-a2e3-dedd7796d92a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:38.577251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.577206    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.577251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.577241    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565adc5d-b9ed-457c-a2e3-dedd7796d92a-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.577251 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.577254    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.577728 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.577263    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.577728 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.577270    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/565adc5d-b9ed-457c-a2e3-dedd7796d92a-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.577728 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.577278    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz8jj\" (UniqueName: \"kubernetes.io/projected/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kube-api-access-mz8jj\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:42:38.577728 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:38.577287    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565adc5d-b9ed-457c-a2e3-dedd7796d92a-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:42:39.295972 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:39.295936    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"
Sep 16 19:42:39.296154 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:39.295940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49" event={"ID":"565adc5d-b9ed-457c-a2e3-dedd7796d92a","Type":"ContainerDied","Data":"a48c053293830e75ffabb27b133e77eb70140cefc3563f368299b1050f11201c"}
Sep 16 19:42:39.296154 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:39.296088    2561 scope.go:117] "RemoveContainer" containerID="b1959521bda91509f37abdb32df6a10a58bb98a554f4d143b2d7ac12c81c7182"
Sep 16 19:42:39.304893 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:39.304871    2561 scope.go:117] "RemoveContainer" containerID="c0eaf5d94197403c110fcf2db0313f9ddebe040f35c7f7b1b35068e4a368cc53"
Sep 16 19:42:39.316533 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:39.316509    2561 scope.go:117] "RemoveContainer" containerID="90bc41a3c7f8059bc99b171d904596250f322bf2e99de7484d8fefef0b606cbc"
Sep 16 19:42:39.319883 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:39.319855    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"]
Sep 16 19:42:39.327058 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:39.327030    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7b845d6478-ppb49"]
Sep 16 19:42:39.427523 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:39.427480    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" podUID="a2d62190-33b7-4886-a4ed-708ef3a0559a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:40.749824 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:40.749789    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" path="/var/lib/kubelet/pods/565adc5d-b9ed-457c-a2e3-dedd7796d92a/volumes"
Sep 16 19:42:49.427661 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:49.427613    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" podUID="a2d62190-33b7-4886-a4ed-708ef3a0559a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:49.428173 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:49.427759    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:42:59.427527 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:42:59.427485    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" podUID="a2d62190-33b7-4886-a4ed-708ef3a0559a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:07.768639 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:07.768611    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:43:07.840454 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:07.840348    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hj5j\" (UniqueName: \"kubernetes.io/projected/a2d62190-33b7-4886-a4ed-708ef3a0559a-kube-api-access-7hj5j\") pod \"a2d62190-33b7-4886-a4ed-708ef3a0559a\" (UID: \"a2d62190-33b7-4886-a4ed-708ef3a0559a\") "
Sep 16 19:43:07.840454 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:07.840402    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d62190-33b7-4886-a4ed-708ef3a0559a-tls-certs\") pod \"a2d62190-33b7-4886-a4ed-708ef3a0559a\" (UID: \"a2d62190-33b7-4886-a4ed-708ef3a0559a\") "
Sep 16 19:43:07.842546 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:07.842511    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d62190-33b7-4886-a4ed-708ef3a0559a-kube-api-access-7hj5j" (OuterVolumeSpecName: "kube-api-access-7hj5j") pod "a2d62190-33b7-4886-a4ed-708ef3a0559a" (UID: "a2d62190-33b7-4886-a4ed-708ef3a0559a"). InnerVolumeSpecName "kube-api-access-7hj5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:07.842691 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:07.842516    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d62190-33b7-4886-a4ed-708ef3a0559a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2d62190-33b7-4886-a4ed-708ef3a0559a" (UID: "a2d62190-33b7-4886-a4ed-708ef3a0559a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:07.941789 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:07.941751    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hj5j\" (UniqueName: \"kubernetes.io/projected/a2d62190-33b7-4886-a4ed-708ef3a0559a-kube-api-access-7hj5j\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:43:07.941789 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:07.941786    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d62190-33b7-4886-a4ed-708ef3a0559a-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:43:08.427184 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.427138    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2d62190-33b7-4886-a4ed-708ef3a0559a" containerID="c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991" exitCode=0
Sep 16 19:43:08.427368 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.427189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" event={"ID":"a2d62190-33b7-4886-a4ed-708ef3a0559a","Type":"ContainerDied","Data":"c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991"}
Sep 16 19:43:08.427368 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.427209    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"
Sep 16 19:43:08.427368 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.427283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6" event={"ID":"a2d62190-33b7-4886-a4ed-708ef3a0559a","Type":"ContainerDied","Data":"12af725dbc6a85db3e69a3b036960f6b2bf8795d6e0b6334741561edbbce59b7"}
Sep 16 19:43:08.427368 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.427315    2561 scope.go:117] "RemoveContainer" containerID="c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991"
Sep 16 19:43:08.435793 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.435772    2561 scope.go:117] "RemoveContainer" containerID="c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991"
Sep 16 19:43:08.436065 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:43:08.436043    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991\": container with ID starting with c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991 not found: ID does not exist" containerID="c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991"
Sep 16 19:43:08.436120 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.436075    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991"} err="failed to get container status \"c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991\": rpc error: code = NotFound desc = could not find container \"c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991\": container with ID starting with c65abfd3b90593842dc2258f20f47ffd36c04f833e9dfa9cd9a6af16ddad5991 not found: ID does not exist"
Sep 16 19:43:08.451180 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.451152    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"]
Sep 16 19:43:08.454303 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.454279    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48482w6"]
Sep 16 19:43:08.750756 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:08.750718    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d62190-33b7-4886-a4ed-708ef3a0559a" path="/var/lib/kubelet/pods/a2d62190-33b7-4886-a4ed-708ef3a0559a/volumes"
Sep 16 19:43:18.487563 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.487528    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"]
Sep 16 19:43:18.487994 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.487972    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d62190-33b7-4886-a4ed-708ef3a0559a" containerName="main"
Sep 16 19:43:18.488042 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.487997    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d62190-33b7-4886-a4ed-708ef3a0559a" containerName="main"
Sep 16 19:43:18.488042 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.488015    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main"
Sep 16 19:43:18.488042 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.488024    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main"
Sep 16 19:43:18.488042 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.488037    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="storage-initializer"
Sep 16 19:43:18.488042 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.488043    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="storage-initializer"
Sep 16 19:43:18.488229 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.488055    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="llm-d-routing-sidecar"
Sep 16 19:43:18.488229 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.488062    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="llm-d-routing-sidecar"
Sep 16 19:43:18.488229 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.488118    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="llm-d-routing-sidecar"
Sep 16 19:43:18.488229 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.488127    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d62190-33b7-4886-a4ed-708ef3a0559a" containerName="main"
Sep 16 19:43:18.488229 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.488133    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="565adc5d-b9ed-457c-a2e3-dedd7796d92a" containerName="main"
Sep 16 19:43:18.492771 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.492747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.495455 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.495423    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:43:18.495626 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.495501    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:43:18.495626 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.495525    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-w5tnr\""
Sep 16 19:43:18.495626 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.495542    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:43:18.495626 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.495531    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:43:18.495838 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.495635    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:43:18.496009 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.495990    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-94r2g\""
Sep 16 19:43:18.512098 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.512067    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"]
Sep 16 19:43:18.637198 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.637138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.637198 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.637204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-model-cache\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.637436 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.637229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tls-certs\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.637436 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.637272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjdr\" (UniqueName: \"kubernetes.io/projected/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kube-api-access-7pjdr\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.637436 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.637291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.637436 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.637313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-dshm\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.637436 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.637373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-home\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.637436 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.637419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.738745 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.738642    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-home\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.738745 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.738694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.738745 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.738729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.739055 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.738759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-model-cache\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.739055 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.738778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tls-certs\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.739055 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.738814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjdr\" (UniqueName: \"kubernetes.io/projected/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kube-api-access-7pjdr\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.739055 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.738839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.739055 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.738875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-dshm\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.739312 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.739146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-home\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.739312 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.739197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-model-cache\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.739312 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.739219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.739425 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.739308    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.741219 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.741194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-dshm\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.741219 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.741212    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:43:18.742040 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.742022    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:43:18.748199 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.748171    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:43:18.749555 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.749529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.751305 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.751282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tls-certs\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.759151 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.759130    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:43:18.770478 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.770439    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjdr\" (UniqueName: \"kubernetes.io/projected/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kube-api-access-7pjdr\") pod \"router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.811919 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.811883    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-94r2g\""
Sep 16 19:43:18.815911 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.815883    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-w5tnr\""
Sep 16 19:43:18.824799 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.824760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:18.932097 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.932049    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"]
Sep 16 19:43:18.935660 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.935635    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:18.938461 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.938441    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-h7xb7\""
Sep 16 19:43:18.947079 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:18.947050    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"]
Sep 16 19:43:19.042045 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.041932    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n\" (UID: \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:19.042045 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.042018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tds7\" (UniqueName: \"kubernetes.io/projected/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-kube-api-access-2tds7\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n\" (UID: \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:19.110098 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.110065    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"]
Sep 16 19:43:19.112651 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:43:19.112620    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9235c3e_9e93_4b71_8ff8_2360fe3117a0.slice/crio-ce455ead1383bbab8314fd07302417a2161bbb0df0e207afce7c5dc016e360db WatchSource:0}: Error finding container ce455ead1383bbab8314fd07302417a2161bbb0df0e207afce7c5dc016e360db: Status 404 returned error can't find the container with id ce455ead1383bbab8314fd07302417a2161bbb0df0e207afce7c5dc016e360db
Sep 16 19:43:19.142602 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.142552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n\" (UID: \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:19.142732 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.142631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tds7\" (UniqueName: \"kubernetes.io/projected/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-kube-api-access-2tds7\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n\" (UID: \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:19.144985 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.144962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n\" (UID: \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:19.151346 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.151310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tds7\" (UniqueName: \"kubernetes.io/projected/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-kube-api-access-2tds7\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n\" (UID: \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:19.248015 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.247971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:19.482411 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.482371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" event={"ID":"d9235c3e-9e93-4b71-8ff8-2360fe3117a0","Type":"ContainerStarted","Data":"9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6"}
Sep 16 19:43:19.482622 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.482427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" event={"ID":"d9235c3e-9e93-4b71-8ff8-2360fe3117a0","Type":"ContainerStarted","Data":"ce455ead1383bbab8314fd07302417a2161bbb0df0e207afce7c5dc016e360db"}
Sep 16 19:43:19.482622 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.482449    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:19.485721 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.485694    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:43:19.520834 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:19.520808    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"]
Sep 16 19:43:19.525737 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:43:19.525707    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd790d2fe_c5a8_4088_a3b5_779e7f1d98f3.slice/crio-95e3254c0062b77ed87c2762398d5f6cc088bafce24efd68f4eae55531a5b5e9 WatchSource:0}: Error finding container 95e3254c0062b77ed87c2762398d5f6cc088bafce24efd68f4eae55531a5b5e9: Status 404 returned error can't find the container with id 95e3254c0062b77ed87c2762398d5f6cc088bafce24efd68f4eae55531a5b5e9
Sep 16 19:43:20.488629 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:20.488585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" event={"ID":"d9235c3e-9e93-4b71-8ff8-2360fe3117a0","Type":"ContainerStarted","Data":"9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29"}
Sep 16 19:43:20.490122 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:20.490082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" event={"ID":"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3","Type":"ContainerStarted","Data":"7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426"}
Sep 16 19:43:20.490241 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:20.490133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" event={"ID":"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3","Type":"ContainerStarted","Data":"95e3254c0062b77ed87c2762398d5f6cc088bafce24efd68f4eae55531a5b5e9"}
Sep 16 19:43:20.490241 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:20.490173    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:20.531726 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:20.531672    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" podStartSLOduration=2.531654524 podStartE2EDuration="2.531654524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:20.530993179 +0000 UTC m=+3842.319321546" watchObservedRunningTime="2026-09-16 19:43:20.531654524 +0000 UTC m=+3842.319982890"
Sep 16 19:43:22.504418 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:22.504378    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerID="9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29" exitCode=0
Sep 16 19:43:22.504831 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:22.504448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" event={"ID":"d9235c3e-9e93-4b71-8ff8-2360fe3117a0","Type":"ContainerDied","Data":"9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29"}
Sep 16 19:43:23.512798 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:23.512760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" event={"ID":"d9235c3e-9e93-4b71-8ff8-2360fe3117a0","Type":"ContainerStarted","Data":"c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a"}
Sep 16 19:43:23.541263 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:23.541200    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podStartSLOduration=3.288116795 podStartE2EDuration="5.541177528s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.253060733s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:23.53847307 +0000 UTC m=+3845.326801450" watchObservedRunningTime="2026-09-16 19:43:23.541177528 +0000 UTC m=+3845.329505893"
Sep 16 19:43:28.825493 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:28.825447    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:28.825920 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:28.825523    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:28.826883 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:28.826854    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:43:38.825690 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:38.825638    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:43:38.839073 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:38.839039    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:43:48.825509 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:48.825398    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:43:51.499991 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:51.499957    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:43:58.825855 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:43:58.825800    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:44:08.826258 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:44:08.826200    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:44:18.826197 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:44:18.826143    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:44:19.007945 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:44:19.007912    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:44:19.015094 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:44:19.015066    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:44:28.825325 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:44:28.825270    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:44:38.825842 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:44:38.825796    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:44:48.825722 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:44:48.825674    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:44:58.825698 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:44:58.825641    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:45:08.826196 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:08.826141    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Sep 16 19:45:18.835652 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:18.835556    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:45:18.847829 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:18.847792    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:45:38.700083 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:38.700042    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"]
Sep 16 19:45:38.704702 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:38.704670    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"]
Sep 16 19:45:41.500481 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:41.500434    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:51.499674 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:51.499628    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:53.722259 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:53.722214    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerName="main" containerID="cri-o://7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426" gracePeriod=45
Sep 16 19:45:53.730048 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:53.730001    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main" containerID="cri-o://c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a" gracePeriod=45
Sep 16 19:45:54.342042 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.341992    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="llm-d-routing-sidecar" containerID="cri-o://9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6" gracePeriod=45
Sep 16 19:45:54.598456 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.598428    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:45:54.670747 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.670711    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tls-certs\") pod \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") "
Sep 16 19:45:54.670918 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.670772    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-dshm\") pod \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") "
Sep 16 19:45:54.670918 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.670812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kserve-provision-location\") pod \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") "
Sep 16 19:45:54.670918 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.670847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-home\") pod \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") "
Sep 16 19:45:54.670918 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.670876    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pjdr\" (UniqueName: \"kubernetes.io/projected/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kube-api-access-7pjdr\") pod \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") "
Sep 16 19:45:54.670918 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.670900    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tmp-dir\") pod \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") "
Sep 16 19:45:54.671215 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.670926    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-cabundle-cert\") pod \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") "
Sep 16 19:45:54.671215 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.671016    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-model-cache\") pod \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\" (UID: \"d9235c3e-9e93-4b71-8ff8-2360fe3117a0\") "
Sep 16 19:45:54.671464 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.671418    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9235c3e-9e93-4b71-8ff8-2360fe3117a0" (UID: "d9235c3e-9e93-4b71-8ff8-2360fe3117a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:45:54.671623 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.671468    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9235c3e-9e93-4b71-8ff8-2360fe3117a0" (UID: "d9235c3e-9e93-4b71-8ff8-2360fe3117a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:54.671715 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.671677    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-home" (OuterVolumeSpecName: "home") pod "d9235c3e-9e93-4b71-8ff8-2360fe3117a0" (UID: "d9235c3e-9e93-4b71-8ff8-2360fe3117a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:54.673765 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.673702    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-dshm" (OuterVolumeSpecName: "dshm") pod "d9235c3e-9e93-4b71-8ff8-2360fe3117a0" (UID: "d9235c3e-9e93-4b71-8ff8-2360fe3117a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:54.673765 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.673733    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kube-api-access-7pjdr" (OuterVolumeSpecName: "kube-api-access-7pjdr") pod "d9235c3e-9e93-4b71-8ff8-2360fe3117a0" (UID: "d9235c3e-9e93-4b71-8ff8-2360fe3117a0"). InnerVolumeSpecName "kube-api-access-7pjdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:54.673765 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.673732    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9235c3e-9e93-4b71-8ff8-2360fe3117a0" (UID: "d9235c3e-9e93-4b71-8ff8-2360fe3117a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:54.683513 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.683478    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9235c3e-9e93-4b71-8ff8-2360fe3117a0" (UID: "d9235c3e-9e93-4b71-8ff8-2360fe3117a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:54.695241 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.695203    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9235c3e-9e93-4b71-8ff8-2360fe3117a0" (UID: "d9235c3e-9e93-4b71-8ff8-2360fe3117a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:54.771795 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.771756    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:45:54.772212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.771895    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:45:54.772212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.771928    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pjdr\" (UniqueName: \"kubernetes.io/projected/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-kube-api-access-7pjdr\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:45:54.772212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.771946    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:45:54.772212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.771961    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:45:54.772212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.771982    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:45:54.772212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.771994    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:45:54.772212 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:54.772001    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9235c3e-9e93-4b71-8ff8-2360fe3117a0-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:45:55.250147 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.250106    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerID="c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a" exitCode=0
Sep 16 19:45:55.250147 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.250137    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerID="9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6" exitCode=0
Sep 16 19:45:55.250374 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.250173    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"
Sep 16 19:45:55.250374 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.250184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" event={"ID":"d9235c3e-9e93-4b71-8ff8-2360fe3117a0","Type":"ContainerDied","Data":"c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a"}
Sep 16 19:45:55.250374 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.250243    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" event={"ID":"d9235c3e-9e93-4b71-8ff8-2360fe3117a0","Type":"ContainerDied","Data":"9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6"}
Sep 16 19:45:55.250374 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.250262    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6" event={"ID":"d9235c3e-9e93-4b71-8ff8-2360fe3117a0","Type":"ContainerDied","Data":"ce455ead1383bbab8314fd07302417a2161bbb0df0e207afce7c5dc016e360db"}
Sep 16 19:45:55.250374 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.250283    2561 scope.go:117] "RemoveContainer" containerID="c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a"
Sep 16 19:45:55.259412 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.259392    2561 scope.go:117] "RemoveContainer" containerID="9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29"
Sep 16 19:45:55.267403 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.267381    2561 scope.go:117] "RemoveContainer" containerID="9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6"
Sep 16 19:45:55.272550 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.272524    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"]
Sep 16 19:45:55.277910 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.277884    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-ff9f64dd6-4g5q6"]
Sep 16 19:45:55.280461 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.280443    2561 scope.go:117] "RemoveContainer" containerID="c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a"
Sep 16 19:45:55.280806 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:45:55.280782    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a\": container with ID starting with c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a not found: ID does not exist" containerID="c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a"
Sep 16 19:45:55.280870 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.280815    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a"} err="failed to get container status \"c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a\": rpc error: code = NotFound desc = could not find container \"c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a\": container with ID starting with c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a not found: ID does not exist"
Sep 16 19:45:55.280870 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.280834    2561 scope.go:117] "RemoveContainer" containerID="9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29"
Sep 16 19:45:55.281125 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:45:55.281105    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29\": container with ID starting with 9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29 not found: ID does not exist" containerID="9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29"
Sep 16 19:45:55.281195 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.281136    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29"} err="failed to get container status \"9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29\": rpc error: code = NotFound desc = could not find container \"9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29\": container with ID starting with 9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29 not found: ID does not exist"
Sep 16 19:45:55.281195 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.281161    2561 scope.go:117] "RemoveContainer" containerID="9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6"
Sep 16 19:45:55.281429 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:45:55.281411    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6\": container with ID starting with 9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6 not found: ID does not exist" containerID="9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6"
Sep 16 19:45:55.281475 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.281434    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6"} err="failed to get container status \"9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6\": rpc error: code = NotFound desc = could not find container \"9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6\": container with ID starting with 9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6 not found: ID does not exist"
Sep 16 19:45:55.281475 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.281449    2561 scope.go:117] "RemoveContainer" containerID="c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a"
Sep 16 19:45:55.281698 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.281677    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a"} err="failed to get container status \"c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a\": rpc error: code = NotFound desc = could not find container \"c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a\": container with ID starting with c7f6b0f6b510ffc5370db292d3de3f1890e70fa7ce69f3235259b882c0ba8f7a not found: ID does not exist"
Sep 16 19:45:55.281762 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.281698    2561 scope.go:117] "RemoveContainer" containerID="9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29"
Sep 16 19:45:55.281917 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.281896    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29"} err="failed to get container status \"9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29\": rpc error: code = NotFound desc = could not find container \"9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29\": container with ID starting with 9dd692a35e15df7fd79de2bb03b4927bfd533e3231707e17305de5bdc6810b29 not found: ID does not exist"
Sep 16 19:45:55.281964 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.281918    2561 scope.go:117] "RemoveContainer" containerID="9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6"
Sep 16 19:45:55.282126 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:55.282107    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6"} err="failed to get container status \"9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6\": rpc error: code = NotFound desc = could not find container \"9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6\": container with ID starting with 9ffef827ca640194318c9d14da15f88e1dd4dab7bd06e7e257fa66c5f68c72f6 not found: ID does not exist"
Sep 16 19:45:56.750030 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:45:56.749997    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" path="/var/lib/kubelet/pods/d9235c3e-9e93-4b71-8ff8-2360fe3117a0/volumes"
Sep 16 19:46:01.499624 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:01.499551    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:01.500021 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:01.499741    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:46:11.499733 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:11.499685    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:21.499459 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:21.499418    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:23.966354 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:23.966325    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:46:24.036357 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.036250    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tds7\" (UniqueName: \"kubernetes.io/projected/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-kube-api-access-2tds7\") pod \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\" (UID: \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\") "
Sep 16 19:46:24.036357 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.036342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-tls-certs\") pod \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\" (UID: \"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3\") "
Sep 16 19:46:24.038500 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.038464    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" (UID: "d790d2fe-c5a8-4088-a3b5-779e7f1d98f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:46:24.038631 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.038542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-kube-api-access-2tds7" (OuterVolumeSpecName: "kube-api-access-2tds7") pod "d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" (UID: "d790d2fe-c5a8-4088-a3b5-779e7f1d98f3"). InnerVolumeSpecName "kube-api-access-2tds7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:24.137440 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.137396    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tds7\" (UniqueName: \"kubernetes.io/projected/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-kube-api-access-2tds7\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:46:24.137440 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.137433    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:46:24.382234 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.382137    2561 generic.go:356] "Generic (PLEG): container finished" podID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerID="7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426" exitCode=0
Sep 16 19:46:24.382234 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.382186    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"
Sep 16 19:46:24.382234 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.382197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" event={"ID":"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3","Type":"ContainerDied","Data":"7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426"}
Sep 16 19:46:24.382500 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.382262    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n" event={"ID":"d790d2fe-c5a8-4088-a3b5-779e7f1d98f3","Type":"ContainerDied","Data":"95e3254c0062b77ed87c2762398d5f6cc088bafce24efd68f4eae55531a5b5e9"}
Sep 16 19:46:24.382500 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.382283    2561 scope.go:117] "RemoveContainer" containerID="7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426"
Sep 16 19:46:24.391785 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.391762    2561 scope.go:117] "RemoveContainer" containerID="7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426"
Sep 16 19:46:24.392051 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:46:24.392029    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426\": container with ID starting with 7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426 not found: ID does not exist" containerID="7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426"
Sep 16 19:46:24.392098 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.392060    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426"} err="failed to get container status \"7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426\": rpc error: code = NotFound desc = could not find container \"7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426\": container with ID starting with 7aea487ef7b330a6032714719ba795a727fd04703c8d6fc94953e671f71e0426 not found: ID does not exist"
Sep 16 19:46:24.406727 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.406698    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"]
Sep 16 19:46:24.414859 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.414836    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45ww42n"]
Sep 16 19:46:24.749240 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:24.749205    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" path="/var/lib/kubelet/pods/d790d2fe-c5a8-4088-a3b5-779e7f1d98f3/volumes"
Sep 16 19:46:36.491792 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.491751    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"]
Sep 16 19:46:36.492182 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492078    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerName="main"
Sep 16 19:46:36.492182 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492090    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerName="main"
Sep 16 19:46:36.492182 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492113    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="storage-initializer"
Sep 16 19:46:36.492182 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492121    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="storage-initializer"
Sep 16 19:46:36.492182 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492130    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main"
Sep 16 19:46:36.492182 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492136    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main"
Sep 16 19:46:36.492182 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492144    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="llm-d-routing-sidecar"
Sep 16 19:46:36.492182 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492149    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="llm-d-routing-sidecar"
Sep 16 19:46:36.492445 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492200    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="llm-d-routing-sidecar"
Sep 16 19:46:36.492445 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492211    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9235c3e-9e93-4b71-8ff8-2360fe3117a0" containerName="main"
Sep 16 19:46:36.492445 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.492218    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d790d2fe-c5a8-4088-a3b5-779e7f1d98f3" containerName="main"
Sep 16 19:46:36.496936 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.496912    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.503651 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.503469    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 19:46:36.503651 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.503556    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 19:46:36.503810 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.503690    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:46:36.505138 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.505112    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 19:46:36.505275 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.505243    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 19:46:36.505347 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.505243    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-dh48s\""
Sep 16 19:46:36.522609 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.522550    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"]
Sep 16 19:46:36.650183 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.650142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-home\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.650183 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.650182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tmp-dir\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.650404 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.650199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.650404 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.650277    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-model-cache\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.650404 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.650320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-cabundle-cert\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.650404 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.650342    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfd86\" (UniqueName: \"kubernetes.io/projected/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kube-api-access-sfd86\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.650404 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.650369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tls-certs\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.650604 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.650432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-dshm\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.750401 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.750324    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"]
Sep 16 19:46:36.751607 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.751585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-home\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.751665 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.751619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tmp-dir\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.751665 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.751636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.751734 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.751668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-model-cache\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.751734 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.751701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-cabundle-cert\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.751734 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.751725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfd86\" (UniqueName: \"kubernetes.io/projected/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kube-api-access-sfd86\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.751852 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.751763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tls-certs\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.751852 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.751811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-dshm\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.752061 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.752032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tmp-dir\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.752160 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.752057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-home\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.752160 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.752136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-model-cache\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.752269 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.752166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.752428 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.752376    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-cabundle-cert\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.754031 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.754007    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:46:36.754134 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.754103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-dshm\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.754427 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.754408    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tls-certs\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.756939 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.756923    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-hc9fk\""
Sep 16 19:46:36.762281 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.762258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfd86\" (UniqueName: \"kubernetes.io/projected/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kube-api-access-sfd86\") pod \"prestop-hook-test-kserve-6f76f59c4d-dm5kf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.764957 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.764938    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"]
Sep 16 19:46:36.807649 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.807616    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:36.853040 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.853009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8151505-2b3b-4c37-a7f5-5c253240957b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd\" (UID: \"c8151505-2b3b-4c37-a7f5-5c253240957b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:46:36.853233 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.853164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rt8q\" (UniqueName: \"kubernetes.io/projected/c8151505-2b3b-4c37-a7f5-5c253240957b-kube-api-access-5rt8q\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd\" (UID: \"c8151505-2b3b-4c37-a7f5-5c253240957b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:46:36.954587 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.954524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rt8q\" (UniqueName: \"kubernetes.io/projected/c8151505-2b3b-4c37-a7f5-5c253240957b-kube-api-access-5rt8q\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd\" (UID: \"c8151505-2b3b-4c37-a7f5-5c253240957b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:46:36.954771 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.954597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8151505-2b3b-4c37-a7f5-5c253240957b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd\" (UID: \"c8151505-2b3b-4c37-a7f5-5c253240957b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:46:36.957086 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.957057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8151505-2b3b-4c37-a7f5-5c253240957b-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd\" (UID: \"c8151505-2b3b-4c37-a7f5-5c253240957b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:46:36.966840 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:36.966812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rt8q\" (UniqueName: \"kubernetes.io/projected/c8151505-2b3b-4c37-a7f5-5c253240957b-kube-api-access-5rt8q\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd\" (UID: \"c8151505-2b3b-4c37-a7f5-5c253240957b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:46:37.082776 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:37.082690    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:46:37.093559 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:37.093530    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"]
Sep 16 19:46:37.094810 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:46:37.094787    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f0ff00_d84c_4eb2_9ff0_c4da62913ddf.slice/crio-a92ddc1c22815db9152abc1c385b6bef75d4c283c5ae60b06c6c1547fccbe321 WatchSource:0}: Error finding container a92ddc1c22815db9152abc1c385b6bef75d4c283c5ae60b06c6c1547fccbe321: Status 404 returned error can't find the container with id a92ddc1c22815db9152abc1c385b6bef75d4c283c5ae60b06c6c1547fccbe321
Sep 16 19:46:37.360423 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:37.360338    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"]
Sep 16 19:46:37.363975 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:46:37.363948    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8151505_2b3b_4c37_a7f5_5c253240957b.slice/crio-8ac68ca9a609b5f99d81e1744494049cc800cb60abd9a1b59d1f42e5ba5cb0ec WatchSource:0}: Error finding container 8ac68ca9a609b5f99d81e1744494049cc800cb60abd9a1b59d1f42e5ba5cb0ec: Status 404 returned error can't find the container with id 8ac68ca9a609b5f99d81e1744494049cc800cb60abd9a1b59d1f42e5ba5cb0ec
Sep 16 19:46:37.444343 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:37.444282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" event={"ID":"c8151505-2b3b-4c37-a7f5-5c253240957b","Type":"ContainerStarted","Data":"ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34"}
Sep 16 19:46:37.444538 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:37.444392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" event={"ID":"c8151505-2b3b-4c37-a7f5-5c253240957b","Type":"ContainerStarted","Data":"8ac68ca9a609b5f99d81e1744494049cc800cb60abd9a1b59d1f42e5ba5cb0ec"}
Sep 16 19:46:37.444538 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:37.444428    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:46:37.445966 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:37.445940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" event={"ID":"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf","Type":"ContainerStarted","Data":"90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933"}
Sep 16 19:46:37.446092 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:37.445982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" event={"ID":"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf","Type":"ContainerStarted","Data":"a92ddc1c22815db9152abc1c385b6bef75d4c283c5ae60b06c6c1547fccbe321"}
Sep 16 19:46:37.464047 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:37.463967    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" podStartSLOduration=1.463945948 podStartE2EDuration="1.463945948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:37.462215493 +0000 UTC m=+4039.250543881" watchObservedRunningTime="2026-09-16 19:46:37.463945948 +0000 UTC m=+4039.252274315"
Sep 16 19:46:39.459546 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:39.459508    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerID="90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933" exitCode=0
Sep 16 19:46:39.459961 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:39.459593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" event={"ID":"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf","Type":"ContainerDied","Data":"90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933"}
Sep 16 19:46:40.467065 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:40.467027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" event={"ID":"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf","Type":"ContainerStarted","Data":"0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0"}
Sep 16 19:46:40.494232 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:40.494168    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podStartSLOduration=2.235076273 podStartE2EDuration="4.494143775s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.259067502s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:40.492546089 +0000 UTC m=+4042.280874457" watchObservedRunningTime="2026-09-16 19:46:40.494143775 +0000 UTC m=+4042.282472153"
Sep 16 19:46:40.808113 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:40.808069    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:46.808463 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:46.808417    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:46:46.810177 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:46.810145    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:46:56.809093 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:46:56.809038    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:47:06.808393 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:47:06.808339    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:47:08.455764 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:47:08.455724    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:47:16.808538 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:47:16.808486    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:47:26.808897 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:47:26.808846    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:47:36.808207 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:47:36.808150    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:47:46.808472 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:47:46.808423    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:47:56.808524 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:47:56.808474    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:48:06.808420 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:06.808374    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:48:16.808748 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:16.808645    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:48:26.808959 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:26.808902    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:48:36.817950 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:36.817918    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:48:36.825728 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:36.825701    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:48:39.756134 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:39.756093    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"]
Sep 16 19:48:54.778681 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:54.778601    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" containerID="cri-o://0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0" gracePeriod=45
Sep 16 19:48:54.818668 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:54.818617    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 16 19:48:55.522816 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.522785    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:48:55.609391 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.609309    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfd86\" (UniqueName: \"kubernetes.io/projected/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kube-api-access-sfd86\") pod \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") "
Sep 16 19:48:55.609391 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.609351    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-model-cache\") pod \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") "
Sep 16 19:48:55.609391 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.609389    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kserve-provision-location\") pod \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") "
Sep 16 19:48:55.609697 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.609405    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tmp-dir\") pod \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") "
Sep 16 19:48:55.609697 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.609450    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-home\") pod \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") "
Sep 16 19:48:55.609697 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.609473    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-cabundle-cert\") pod \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") "
Sep 16 19:48:55.609697 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.609495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-dshm\") pod \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") "
Sep 16 19:48:55.609892 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.609708    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" (UID: "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:55.609892 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.609880    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:48:55.610398 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.610366    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" (UID: "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:48:55.610398 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.610369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-home" (OuterVolumeSpecName: "home") pod "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" (UID: "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:55.611856 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.611823    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kube-api-access-sfd86" (OuterVolumeSpecName: "kube-api-access-sfd86") pod "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" (UID: "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf"). InnerVolumeSpecName "kube-api-access-sfd86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:55.612090 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.612067    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-dshm" (OuterVolumeSpecName: "dshm") pod "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" (UID: "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:55.621942 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.621900    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" (UID: "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:55.636011 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.635979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" (UID: "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:55.710399 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.710363    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tls-certs\") pod \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\" (UID: \"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf\") "
Sep 16 19:48:55.710636 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.710623    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:48:55.710719 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.710644    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:48:55.710719 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.710658    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:48:55.710719 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.710671    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:48:55.710719 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.710684    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:48:55.710719 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.710695    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfd86\" (UniqueName: \"kubernetes.io/projected/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-kube-api-access-sfd86\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:48:55.712494 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.712469    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" (UID: "f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:48:55.811156 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:55.811119    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:48:56.125042 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.125004    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerID="0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0" exitCode=0
Sep 16 19:48:56.125246 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.125063    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"
Sep 16 19:48:56.125246 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.125088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" event={"ID":"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf","Type":"ContainerDied","Data":"0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0"}
Sep 16 19:48:56.125246 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.125146    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf" event={"ID":"f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf","Type":"ContainerDied","Data":"a92ddc1c22815db9152abc1c385b6bef75d4c283c5ae60b06c6c1547fccbe321"}
Sep 16 19:48:56.125246 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.125170    2561 scope.go:117] "RemoveContainer" containerID="0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0"
Sep 16 19:48:56.134097 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.134086    2561 scope.go:117] "RemoveContainer" containerID="90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933"
Sep 16 19:48:56.141660 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.141643    2561 scope.go:117] "RemoveContainer" containerID="0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0"
Sep 16 19:48:56.141942 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:48:56.141923    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0\": container with ID starting with 0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0 not found: ID does not exist" containerID="0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0"
Sep 16 19:48:56.142000 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.141952    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0"} err="failed to get container status \"0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0\": rpc error: code = NotFound desc = could not find container \"0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0\": container with ID starting with 0a6ce2a6e142577f27c164705651d4cb0f977473a91ece5fc1de7591a30511d0 not found: ID does not exist"
Sep 16 19:48:56.142000 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.141969    2561 scope.go:117] "RemoveContainer" containerID="90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933"
Sep 16 19:48:56.142198 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:48:56.142181    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933\": container with ID starting with 90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933 not found: ID does not exist" containerID="90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933"
Sep 16 19:48:56.142244 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.142204    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933"} err="failed to get container status \"90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933\": rpc error: code = NotFound desc = could not find container \"90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933\": container with ID starting with 90700699fadbad489deb0352cf1f027880cec3472ae88c9736cd32266476d933 not found: ID does not exist"
Sep 16 19:48:56.149448 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.149424    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"]
Sep 16 19:48:56.155560 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.155527    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f76f59c4d-dm5kf"]
Sep 16 19:48:56.748922 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:48:56.748884    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" path="/var/lib/kubelet/pods/f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf/volumes"
Sep 16 19:49:06.080623 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:06.080583    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"]
Sep 16 19:49:08.455563 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:08.455518    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:18.455147 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:18.455102    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:19.037694 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:19.037661    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:49:19.045146 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:19.045119    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:49:21.100126 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:21.100079    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerName="main" containerID="cri-o://ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34" gracePeriod=45
Sep 16 19:49:28.455276 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:28.455231    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:28.455789 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:28.455341    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:49:38.455711 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:38.455663    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:48.455729 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:48.455639    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:51.356734 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.356705    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:49:51.373396 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.373314    2561 generic.go:356] "Generic (PLEG): container finished" podID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerID="ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34" exitCode=0
Sep 16 19:49:51.373396 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.373355    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"
Sep 16 19:49:51.373396 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.373364    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" event={"ID":"c8151505-2b3b-4c37-a7f5-5c253240957b","Type":"ContainerDied","Data":"ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34"}
Sep 16 19:49:51.373693 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.373407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd" event={"ID":"c8151505-2b3b-4c37-a7f5-5c253240957b","Type":"ContainerDied","Data":"8ac68ca9a609b5f99d81e1744494049cc800cb60abd9a1b59d1f42e5ba5cb0ec"}
Sep 16 19:49:51.373693 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.373429    2561 scope.go:117] "RemoveContainer" containerID="ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34"
Sep 16 19:49:51.381435 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.381417    2561 scope.go:117] "RemoveContainer" containerID="ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34"
Sep 16 19:49:51.381719 ip-10-0-142-197 kubenswrapper[2561]: E0916 19:49:51.381700    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34\": container with ID starting with ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34 not found: ID does not exist" containerID="ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34"
Sep 16 19:49:51.381793 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.381733    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34"} err="failed to get container status \"ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34\": rpc error: code = NotFound desc = could not find container \"ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34\": container with ID starting with ea11144044a51d970d5a7ed3055d8fec922df16f9c54bca1d59197ac164e2a34 not found: ID does not exist"
Sep 16 19:49:51.387792 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.387767    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8151505-2b3b-4c37-a7f5-5c253240957b-tls-certs\") pod \"c8151505-2b3b-4c37-a7f5-5c253240957b\" (UID: \"c8151505-2b3b-4c37-a7f5-5c253240957b\") "
Sep 16 19:49:51.387898 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.387824    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rt8q\" (UniqueName: \"kubernetes.io/projected/c8151505-2b3b-4c37-a7f5-5c253240957b-kube-api-access-5rt8q\") pod \"c8151505-2b3b-4c37-a7f5-5c253240957b\" (UID: \"c8151505-2b3b-4c37-a7f5-5c253240957b\") "
Sep 16 19:49:51.390071 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.390041    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8151505-2b3b-4c37-a7f5-5c253240957b-kube-api-access-5rt8q" (OuterVolumeSpecName: "kube-api-access-5rt8q") pod "c8151505-2b3b-4c37-a7f5-5c253240957b" (UID: "c8151505-2b3b-4c37-a7f5-5c253240957b"). InnerVolumeSpecName "kube-api-access-5rt8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:51.390168 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.390100    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8151505-2b3b-4c37-a7f5-5c253240957b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8151505-2b3b-4c37-a7f5-5c253240957b" (UID: "c8151505-2b3b-4c37-a7f5-5c253240957b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:51.488927 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.488887    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8151505-2b3b-4c37-a7f5-5c253240957b-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:49:51.488927 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.488919    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rt8q\" (UniqueName: \"kubernetes.io/projected/c8151505-2b3b-4c37-a7f5-5c253240957b-kube-api-access-5rt8q\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 16 19:49:51.707406 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.707369    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"]
Sep 16 19:49:51.710846 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:51.710817    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-frrwd"]
Sep 16 19:49:52.749147 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:49:52.749108    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" path="/var/lib/kubelet/pods/c8151505-2b3b-4c37-a7f5-5c253240957b/volumes"
Sep 16 19:50:26.956719 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:26.956681    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-mwk6z_659946bd-e2e2-4aed-95f0-5aeaeab3e745/manager/0.log"
Sep 16 19:50:26.962159 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:26.962137    2561 ???:1] "http: TLS handshake error from 10.0.130.204:60636: EOF"
Sep 16 19:50:27.169561 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:27.169526    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-f9pfb_735d7064-203c-4a76-acba-2912d2ab58bf/limitador/0.log"
Sep 16 19:50:36.509815 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:36.509780    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ztbzx_2447bda6-429c-48a3-ad0a-e8b58294e883/global-pull-secret-syncer/0.log"
Sep 16 19:50:36.600088 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:36.600051    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pvshm_fa398820-2005-47bc-8687-b89d132bb6fb/konnectivity-agent/0.log"
Sep 16 19:50:36.705347 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:36.705305    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-197.ec2.internal_aa12150102fe3893c84359d7f70bc6ce/haproxy/0.log"
Sep 16 19:50:42.258223 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:42.258182    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-mwk6z_659946bd-e2e2-4aed-95f0-5aeaeab3e745/manager/0.log"
Sep 16 19:50:42.518306 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:42.518224    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-f9pfb_735d7064-203c-4a76-acba-2912d2ab58bf/limitador/0.log"
Sep 16 19:50:44.166458 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.166417    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-hc2fz_5a1a36bd-4778-4ef6-8bc8-6894080ed305/kube-state-metrics/0.log"
Sep 16 19:50:44.192343 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.192309    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-hc2fz_5a1a36bd-4778-4ef6-8bc8-6894080ed305/kube-rbac-proxy-main/0.log"
Sep 16 19:50:44.231784 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.231743    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-hc2fz_5a1a36bd-4778-4ef6-8bc8-6894080ed305/kube-rbac-proxy-self/0.log"
Sep 16 19:50:44.276504 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.276420    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5455b4cb7d-64f7h_5f926833-0910-4be6-a32c-6ffb709b8041/metrics-server/0.log"
Sep 16 19:50:44.306558 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.306526    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-hp5qq_b5bf780c-04ef-48e4-9fee-aad7d925af23/monitoring-plugin/0.log"
Sep 16 19:50:44.429670 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.429631    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h95tc_4701e0a7-c118-47b8-9ff7-d38c38a7794d/node-exporter/0.log"
Sep 16 19:50:44.451472 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.451441    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h95tc_4701e0a7-c118-47b8-9ff7-d38c38a7794d/kube-rbac-proxy/0.log"
Sep 16 19:50:44.474357 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.474323    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h95tc_4701e0a7-c118-47b8-9ff7-d38c38a7794d/init-textfile/0.log"
Sep 16 19:50:44.711353 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.711326    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3de4adda-c570-4c43-87b8-65228d5d0a32/prometheus/0.log"
Sep 16 19:50:44.731440 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.731395    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3de4adda-c570-4c43-87b8-65228d5d0a32/config-reloader/0.log"
Sep 16 19:50:44.757599 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.755974    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3de4adda-c570-4c43-87b8-65228d5d0a32/thanos-sidecar/0.log"
Sep 16 19:50:44.779477 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.779444    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3de4adda-c570-4c43-87b8-65228d5d0a32/kube-rbac-proxy-web/0.log"
Sep 16 19:50:44.807703 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.807673    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3de4adda-c570-4c43-87b8-65228d5d0a32/kube-rbac-proxy/0.log"
Sep 16 19:50:44.833278 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.833227    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3de4adda-c570-4c43-87b8-65228d5d0a32/kube-rbac-proxy-thanos/0.log"
Sep 16 19:50:44.858229 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.858201    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3de4adda-c570-4c43-87b8-65228d5d0a32/init-config-reloader/0.log"
Sep 16 19:50:44.895605 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.895557    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-nl5x9_1cd53286-79c3-4788-b928-e822171d87ab/prometheus-operator/0.log"
Sep 16 19:50:44.916500 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:44.916417    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-nl5x9_1cd53286-79c3-4788-b928-e822171d87ab/kube-rbac-proxy/0.log"
Sep 16 19:50:45.066591 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:45.066533    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d8d5f8d4-vhcbs_c8c40772-65e2-48bf-be54-71ce2842fc50/thanos-query/0.log"
Sep 16 19:50:45.086537 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:45.086485    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d8d5f8d4-vhcbs_c8c40772-65e2-48bf-be54-71ce2842fc50/kube-rbac-proxy-web/0.log"
Sep 16 19:50:45.108211 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:45.108185    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d8d5f8d4-vhcbs_c8c40772-65e2-48bf-be54-71ce2842fc50/kube-rbac-proxy/0.log"
Sep 16 19:50:45.131765 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:45.131732    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d8d5f8d4-vhcbs_c8c40772-65e2-48bf-be54-71ce2842fc50/prom-label-proxy/0.log"
Sep 16 19:50:45.153216 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:45.153190    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d8d5f8d4-vhcbs_c8c40772-65e2-48bf-be54-71ce2842fc50/kube-rbac-proxy-rules/0.log"
Sep 16 19:50:45.174302 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:45.174232    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76d8d5f8d4-vhcbs_c8c40772-65e2-48bf-be54-71ce2842fc50/kube-rbac-proxy-metrics/0.log"
Sep 16 19:50:46.760671 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.760630    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/2.log"
Sep 16 19:50:46.765295 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.765272    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-j9cpz_75109d51-2bb5-4be7-816d-3c4d5b5f9a5d/console-operator/3.log"
Sep 16 19:50:46.964260 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.964219    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"]
Sep 16 19:50:46.964611 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.964597    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main"
Sep 16 19:50:46.964680 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.964613    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main"
Sep 16 19:50:46.964680 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.964625    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="storage-initializer"
Sep 16 19:50:46.964680 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.964633    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="storage-initializer"
Sep 16 19:50:46.964680 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.964646    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerName="main"
Sep 16 19:50:46.964680 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.964651    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerName="main"
Sep 16 19:50:46.964845 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.964707    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f0ff00-d84c-4eb2-9ff0-c4da62913ddf" containerName="main"
Sep 16 19:50:46.964845 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.964715    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8151505-2b3b-4c37-a7f5-5c253240957b" containerName="main"
Sep 16 19:50:46.968044 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.968015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:46.970588 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.970546    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s865n\"/\"default-dockercfg-b7pzr\""
Sep 16 19:50:46.970754 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.970554    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s865n\"/\"openshift-service-ca.crt\""
Sep 16 19:50:46.971473 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.971448    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s865n\"/\"kube-root-ca.crt\""
Sep 16 19:50:46.977074 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:46.977044    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"]
Sep 16 19:50:47.071762 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.071674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-proc\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.071762 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.071707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-podres\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.071949 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.071804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9nf\" (UniqueName: \"kubernetes.io/projected/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-kube-api-access-9l9nf\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.071949 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.071852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-lib-modules\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.071949 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.071918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-sys\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.173216 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.173172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-sys\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.173409 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.173226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-proc\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.173409 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.173315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-podres\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.173409 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.173311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-sys\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.173409 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.173378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-proc\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.173656 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.173424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-podres\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.173656 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.173494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9nf\" (UniqueName: \"kubernetes.io/projected/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-kube-api-access-9l9nf\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.173656 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.173548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-lib-modules\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.173765 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.173692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-lib-modules\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.181665 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.181634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9nf\" (UniqueName: \"kubernetes.io/projected/5d4240bc-b9d2-4325-b65f-dcd12b2235c2-kube-api-access-9l9nf\") pod \"perf-node-gather-daemonset-j958l\" (UID: \"5d4240bc-b9d2-4325-b65f-dcd12b2235c2\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.279744 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.279689    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:47.548203 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.548174    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"]
Sep 16 19:50:47.550900 ip-10-0-142-197 kubenswrapper[2561]: W0916 19:50:47.550865    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5d4240bc_b9d2_4325_b65f_dcd12b2235c2.slice/crio-d3bd5b764c2133245e3721074ce46a31ee51bf30898b5b8f528501a571f0a4a0 WatchSource:0}: Error finding container d3bd5b764c2133245e3721074ce46a31ee51bf30898b5b8f528501a571f0a4a0: Status 404 returned error can't find the container with id d3bd5b764c2133245e3721074ce46a31ee51bf30898b5b8f528501a571f0a4a0
Sep 16 19:50:47.623534 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.623488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l" event={"ID":"5d4240bc-b9d2-4325-b65f-dcd12b2235c2","Type":"ContainerStarted","Data":"d3bd5b764c2133245e3721074ce46a31ee51bf30898b5b8f528501a571f0a4a0"}
Sep 16 19:50:47.923219 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:47.923126    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-pprnw_298c6c47-abf6-43c7-afb0-4068922e8803/volume-data-source-validator/0.log"
Sep 16 19:50:48.629490 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:48.629451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l" event={"ID":"5d4240bc-b9d2-4325-b65f-dcd12b2235c2","Type":"ContainerStarted","Data":"9419a8a48256564e0b859d5f52e934a10befda5f106aefc5d042c93325494269"}
Sep 16 19:50:48.629722 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:48.629609    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:48.646675 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:48.646611    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l" podStartSLOduration=2.646593834 podStartE2EDuration="2.646593834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:48.644812596 +0000 UTC m=+4290.433140962" watchObservedRunningTime="2026-09-16 19:50:48.646593834 +0000 UTC m=+4290.434922200"
Sep 16 19:50:49.018013 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:49.017978    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-glgft_15a235aa-917b-41e2-b01a-6249d530b15e/dns/0.log"
Sep 16 19:50:49.043986 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:49.043951    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-glgft_15a235aa-917b-41e2-b01a-6249d530b15e/kube-rbac-proxy/0.log"
Sep 16 19:50:49.163440 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:49.163409    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rrwk9_46d448d3-6c08-4ec6-9d5d-c62c2635e0c2/dns-node-resolver/0.log"
Sep 16 19:50:49.800715 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:49.800673    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-smzkz_61c92186-ba53-4d0d-8c30-5dd1e3778b30/node-ca/0.log"
Sep 16 19:50:51.597272 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:51.597234    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gr4nw_bce3de21-b15d-4124-af2b-4457300f2fd1/serve-healthcheck-canary/0.log"
Sep 16 19:50:52.203983 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:52.203953    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hknqv_9fa04cde-1113-4aaa-84af-1da4349e4b59/kube-rbac-proxy/0.log"
Sep 16 19:50:52.225749 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:52.225726    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hknqv_9fa04cde-1113-4aaa-84af-1da4349e4b59/exporter/0.log"
Sep 16 19:50:52.247327 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:52.247301    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hknqv_9fa04cde-1113-4aaa-84af-1da4349e4b59/extractor/0.log"
Sep 16 19:50:54.644048 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:54.644019    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-j958l"
Sep 16 19:50:55.550342 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:50:55.550308    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-856c96b694-z59vr_3d8f65db-a55e-4803-9d99-b59827b5a570/manager/0.log"
Sep 16 19:51:04.661792 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:04.661757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khznh_dda38eb1-7c5b-42cb-8557-640253d94a7f/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:51:04.684283 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:04.684250    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khznh_dda38eb1-7c5b-42cb-8557-640253d94a7f/egress-router-binary-copy/0.log"
Sep 16 19:51:04.706785 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:04.706753    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khznh_dda38eb1-7c5b-42cb-8557-640253d94a7f/cni-plugins/0.log"
Sep 16 19:51:04.732217 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:04.732187    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khznh_dda38eb1-7c5b-42cb-8557-640253d94a7f/bond-cni-plugin/0.log"
Sep 16 19:51:04.754209 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:04.754157    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khznh_dda38eb1-7c5b-42cb-8557-640253d94a7f/routeoverride-cni/0.log"
Sep 16 19:51:04.775168 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:04.775139    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khznh_dda38eb1-7c5b-42cb-8557-640253d94a7f/whereabouts-cni-bincopy/0.log"
Sep 16 19:51:04.797155 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:04.797126    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khznh_dda38eb1-7c5b-42cb-8557-640253d94a7f/whereabouts-cni/0.log"
Sep 16 19:51:05.040352 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:05.040316    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n9wqg_0930c4f1-458b-47e4-a122-d1080055cd14/kube-multus/0.log"
Sep 16 19:51:05.107152 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:05.107114    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ds74d_e08562f6-72fa-408c-9d9f-43ce0d4ced85/network-metrics-daemon/0.log"
Sep 16 19:51:05.126396 ip-10-0-142-197 kubenswrapper[2561]: I0916 19:51:05.126365    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ds74d_e08562f6-72fa-408c-9d9f-43ce0d4ced85/kube-rbac-proxy/0.log"